HFU HF Underground

Technical Topics => Software => Topic started by: linkz on May 01, 2018, 0755 UTC

Title: directKiwi project
Post by: linkz on May 01, 2018, 0755 UTC
Hi all

Recently I wrote a piece a software that allows you to quickly connect to KiwiSDR remotes around the world
It is based on python, so you have to install some stuff to run it (it should work on Windows/Linux/MacOS X)

What it is doing:
* fast connect from a list of <available> servers (+/- 200)
* fast switch between remotes (good for signal check)
* audio stream only
* bandwidth + AGC/MGC setting available
* update the listing by parsing kiwisdr.com/public webpage (servers listed are checked via GeoIP + port open check)
* returns the s-meter
* light CPU usage for low-power computers
* default parameters saved in config file

What it is not doing:
* real-time parameters changes
* waterfall not displayed
* and.. almost all the OpenwebRX/web interface can do

I think it could be good for signal check on different remotes, faster than opening multiple openwebRX in your web browser


All information on the project page:

http://81.93.247.141/~linkz/directKiwi/

(feedback, requests appreciated.)


&linkz
Title: Re: directKiwi project
Post by: TheRelayStation on May 01, 2018, 1815 UTC
very good job.
will the idle timeout (configured independently by each server) still apply ?
ive found an easy method to prevent the idle timeout by stopping the page script, this stops all functions except for the incoming audio.
modifying the page script also works but its not as easy and quick as simply stopping it.
this manner allows me to pass the I/Q (6Khz max BW) to software of choice such as "sdr-sharp" for the use of adjustments and plugins, other than the ability of frequency control.
Title: Re: directKiwi project
Post by: linkz on May 03, 2018, 2211 UTC
Hi

I'm not sure the timeout is bypassed by the script but i think so as there is no .js scripts involved
I'm trying to fix an issue that appeared recently, i don't know why yet, but software not running and returning a
" _tkinter.TclError: already exists "
on some setups (64bits) in the python console.

It still works ok here on 32bits Win7, no updates made
python 2.7.15 has been released recently, perhaps it's the problem, I will check later

I also need some help to better code the datastream to audio output on MacOS X, it's pretty laggy the way the arrays are managed at the moment

thanks for the try

&linkz
Title: Re: directKiwi project
Post by: TheRelayStation on May 03, 2018, 2308 UTC
Hi

I'm not sure the timeout is bypassed by the script but i think so as there is no .js scripts involved
I'm trying to fix an issue that appeared recently, i don't know why yet, but software not running and returning a
" _tkinter.TclError: already exists "
on some setups (64bits) in the python console.

It still works ok here on 32bits Win7, no updates made
python 2.7.15 has been released recently, perhaps it's the problem, I will check later

I also need some help to better code the datastream to audio output on MacOS X, it's pretty laggy the way the arrays are managed at the moment

thanks for the try

&linkz
thanks for the interesting software to tinker with.
i will keep you updated with any issues/bugs i discover in the process as well as any additional features i can suggest.
interesting page you have too, linkz.
Title: Re: directKiwi project
Post by: linkz on May 04, 2018, 1751 UTC
Some news

I was able to fix the issue that blocked the code run using python 2.7.15

fixed version 3.62 is now available from the project webpage

And, we test that the KiwiSDR "admin timeout setting" is bypassed by directKiwi, no restrictions applied, so.. no kicks


Regards

&linkz

Title: Re: directKiwi project
Post by: TheRelayStation on May 05, 2018, 1646 UTC
And, we test that the KiwiSDR "admin timeout setting" is bypassed by directKiwi, no restrictions applied, so.. no kicks


Regards

&linkz
that is great though i will delegate the use of no time-out operation of directKiwi on Kiwi SDR's that are not popularly used too often, to avoid angering the host operator and users.
i have done the same method as above in the past when stopping the page script.
thanks !
Title: Re: directKiwi project
Post by: benji on May 24, 2018, 1908 UTC
Hi linkz,

I really do like your python program and it is one of the very few using the kiwiclient interface. After working with it a couple of times a few comments / ideas.

a. S-meter.
Perhaps it may be  better to replace the RSSI S-meter reading by a measure of signal to noise. There are different definitions for SNR. It could be expressed as the decibel value of the ratio of the mean over the standard deviation of the measured signal.

Another way of calculating SNR is found at http://rx.linkfanel.net/snr.html .
Quote from the KiwiSDR  forum :
"Auto Scale" button sets WF max/min by taking one waterfall result, sorting the resulting FFT bins, calling the 50% bin the noise level and the 98% bin the signal level. Then WF min is set to the noise level -10 dB and WF max the signal level +30 dB. Works best when zoomed out so there is a variety of signal levels. The sort/bin idea is from Pierre, who uses a similar idea for SNR scoring at http://rx.linkfanel.net and http://rx.linkfanel.net/snr.html
End Quote

I believe this method is also used to give the reported dB value for WSJT modes. Of course the -10 and + 30 dB are not needed for this purpose. Would be interesting to see that as a time plot together with S-meter readings and look under various QRN and QRN conditions at channels with voice and data transmissions to see if this is a useful to quantify what we hear and perceive as "better audio quality". I guess the kiwiclient is the place to add a SNR computation.


b. Scanning over a limited number of kiwi sites and frequencies
I find myself entering a number of frequencies in the GUI of your directKiwi project and then looking at a number of different kiwis to see how they compare.

Actually if this was automated that would be really nice. Something I was thinking of is have the user enter a list of say 10 frequencies, then select from the list of 200+ kiwis his/her favourite ones (perhaps also 10) to monitor. Scanning thru this table will give a dynamically updated view of the "best" kiwi site and frequency combination. Scanning could be sequential per kiwi ip address or perhaps in parallel with multiple web sockets open at the same time.

A 10 by 10 matrix having for instance frequencies as columns and host kiwi as row could display the latest SNR value. When enhanced with some colour coding akin to a heat map this would look quite very cool. The "best" selection of the moment could further be used to provide audio.
Using kiwclient to interrogate remote kiwis via the internet involves of course some time delay so a realistic refresh rate for such a matrix is probably 30-60 seconds.

Anyway just throwing around some ideas for this already nice directKiwi program.


Best regards, Ben
Title: Re: directKiwi project
Post by: linkz on June 02, 2018, 0759 UTC
Hi Ben

thanks for the ideas

a: I don't think i can do this, as the SNR calculation, if i understand correctely is based on a wide display waterfall analysis and "github.com/dev-zzo/kiwiclient" is unable to get a waterfall.
What i can do is getting from linkfanel or marco (IS0KYB) already calculated SNRs and putting the data in the node listing ?
Anyway i wonder if a SNR calculation based on a fixed bandwidth (max 6kHz) demodulated SSB frequency is easy to do and reliable, i mean it should be difficult to compare the desired AF signals strength (it can be anything, not just like WSPR sigs) with the noise background in such a narrow BW..

b: scanning different remotes for various frequencies in an automated way should be codable... but if SNR can't be calculated, we'll just get s-meter info and (recorded?!) audio, so just a s-meter graph or table as result.. not sure if it can be useful.
BTW, i already planned to add a "favorite" system in the node listing, with checkboxes and basic filtering.
Also if i get the already calculated SNR values of each node as mentionned in the "a" answer, I can also filter the worst remotes from the listing, by adding the setting of "do not display remotes that has SNR below than: " but remember the node listing is not dynamic.


ISSUE: Recently i have noticed the listing had mixed countries and hosts. It is due to a specific remote located in India, for some reasons this node is not displayed correctly in kiwiSDR.com/public summary page (due to missing informations)
That make my update process completely failed, by mixing up the .db with false datas.....

I have fixed that issue in the 3.63 version, by computing that specific node informations manually. That new version is available on the project webpage : http://81.93.247.141/~linkz/directKiwi/


Note: I'm working on adding a real RECORDING button in the GUI. In the early versions I have already had a plugin that can record, but it was just based on pygame and can cause different issues depending on the user client PC settings (mainly the capture device setting) so i have removed it.
Remember this project begun by using "github.com/dev-zzo/kiwiclient" stuff in console, script was originally capable writing streams into a .wav file. I have just modified one or two lines of the code to play the audio instead of writing the .wav
So it will not be difficult to come back to the original behaviour, but simultaneously with audio output in speakers


&linkz



Title: Re: directKiwi project
Post by: benji on June 04, 2018, 0555 UTC
Hi Linkz,

I noted the addition of microkiwi_waterfall.py to Christoph's kiwiclient.git which indeed takes a snapshot of a specified host kiwi. With default arguments it shows the calculation result over the whole 30 Mhz range but can be narrowed to a high zoomlevel. Locally I use it for manually selecting a suitable antenna for a chunk of the spectrum. This "global" SNR gives a pretty good indication for SNR at a certain kiwi like Marco shows at http://sibamanna.duckdns.org/sdr_map.html.

Another indication of SNR could be the dB value of signal mean over standard deviation just for the passband. The idea of suggesting a measure of SNR is an attempt to avoid kiwis with high RSSI readings due high local QRN or QRM end up on top of the list.

As you mentioned for voice signals within the passband of the selected mode it remains to be seen if there is any benefit of having such an SNR measure. Probably best way to find out is to plot this together with RSSI in a real time graph.

To go a step further I guess for digital signals measured BER would be a better qualifier. For speech SNR is not well defined since the power spectral density of a voice signal is highly variable with time and almost as random as the noise itself. There are routines trying to separate speech from noise first such as using VAD (voice activity detection) or syllabic rate detection and then estimate SNR. However this is probably overkill for the present project.

Anyway thank you for coming back to my suggestions and a filter for to exclude the worst remotes will already be a good addition.

Best regards, Ben.
Title: Re: directKiwi project
Post by: benson on February 26, 2019, 1349 UTC
Hello Linkz,

Just to let you know that I enjoy using the new version V4.0 of your DirectKiwi program.  The graphical display makes it even easier to compare and browse around the available Kiwis.

I installed it directly in Python running on a windows machine and one thing I noted is that the square icon for the the nodes is a bit small and sometimes points on the coastline of the base map look quite similar. Perhaps a larger or different symbol might help. I am aware that making it too large will cause trouble in areas with high density of Kiwis. I tried different background maps and symbol colours but not much improvement.

For the "nice to have list" a request to cycle manually through the list of favourites nodes.
When this would be done automatically at an n second interval a mean RSSI for each node could be shown....Anyway thank you for making this intuitive and useful GUI available.

Best regards,Ben.
Title: Re: directKiwi project
Post by: linkz on March 16, 2019, 2148 UTC
Hi Ben

Thanks for the feedback again, the only ones i got so far, in fact I have no idea how many people use the software  :P

For Windows you can enable map zooming by uncommenting line 443 of the python code
Hope it will be better for you but it's a bit bugged as the mouse wheel will also act for the embedded console output.. ah Windows..

I'm not really decided to work on directKiwi anymore, my focus is/was on directTDoA project nowadays (version 4.20 is almost at its best)
BTW, the GUI map display is originally from this project so I've decided to merge some of the codes in directKiwi to make it better.

Regards

&linkz



Title: Re: directKiwi project
Post by: linkz on June 10, 2019, 1953 UTC
Hi all

I have released the v5.00 version of directKiwi python GUI code today

Some months ago, I've told you I will not work on the code anymore but finally I've done some modifications to fit my needs :P

Main changes:
* No more s-meter, yes.. no more. No real interest to me and consuming CPU time for nothing.
* The switch between nodes is now faster as you just have to left-click on the square icons to move from node to node.
* It's still possible to get KiwiSDR informations, connect via browser, check SNR, add/rem favorite/blacklist. To do so, just right-click on node icons.
* The code has been cleaned, commented and some parts were totally re-written.
* The directKiwi.cfg file is now in json format. Note that if you already have a filled config file from old GUI versions (<= v4.00), I've add some code to convert it for v5.00 compatibility, just backup & copy your .cfg in the new v5.00 directory. You'll keep your previous settings then.
* Audio parameters (LP filter/HP filter/AGC/MGC/gain/threshold/slope/decay/hang) are now shown in an unique Tk window.
* You can now change the map icon size
* Map zoom is still desactivated but you can still uncomment lines 379 (for Windows) or 380,381 ( for Linux/MacOS) manually to allow that functionnality if needed.

http://81.93.247.141/~linkz/directKiwi/ (http://81.93.247.141/~linkz/directKiwi/)
https://github.com/llinkz/directKiwi (https://github.com/llinkz/directKiwi)


&linkz
Title: Re: directKiwi project
Post by: linkz on November 09, 2019, 2305 UTC
FYI

directKiwi 6.0 is out.

no more pygame & scipy (for MacOS), sounddevice python module + libsamplerate instead -> works with 20kHz KiwiSDRs + no audio compression set by default
IS0KYB microkiwi_waterfall script added (SNR measurement + waterfall display), IS0KYB SNR website source is not used anymore + code clean up



http://81.93.247.141/~linkz/directKiwi/ (http://81.93.247.141/~linkz/directKiwi/)
https://github.com/llinkz/directKiwi (https://github.com/llinkz/directKiwi)


&linkz
Title: Re: directKiwi project
Post by: Josh on November 09, 2019, 2324 UTC
Perhaps a mode or stand alone one click app to tdoa in keeping with the minimum needed to do the job?
Title: Re: directKiwi project
Post by: linkz on November 10, 2019, 0108 UTC
Done... v6.10

using ?ext=tdoa,lat:yyy,lon:xxx,z:5,<submitted node> URL parameter

Note: The submitted node function will probably not working all the time as I'm not using the same algorithm to define empty "tdoa_id" fields from the nodes


&linkz
Title: Re: directKiwi project
Post by: Josh on November 10, 2019, 2013 UTC
Awsome!
Title: Re: directKiwi project
Post by: linkz on April 14, 2020, 2238 UTC
directKiwi v7.00 is out

Finally had some time to work again on the project

Partial Changelog:
Some modifications on the GUI (in fact it uses directTDoA GUI now, so less full restarts when parameter changes, except for the World map)
Python 3 compatibility
Bug fixed on the map update (always fails because of some modifications on the website sources...)

73
&linkz
Title: Re: directKiwi project
Post by: linkz on May 03, 2020, 0823 UTC
directKiwi v7.10 is out

Partial Changelog:
Restart GUI routine modified
directKiwi.bat & setup.bat added for Windows users
Bug fix that caused the map to move suddenly far away when selecting a node (problem only noticed on Windows OS)
Added functionality to manage the overlapping of icons on the map. Now when you click near a cluster of multiple nodes, a menu will appear and allow you to choose the one you really want (to listen or to display node menu)
Some minor bug fixes

73
&linkz