Locked
Re: MQTT Connection in JMRI
Hi Everyone, ? I am very interested in this conversation.? As a DIYer and C/C++ and Java hobbyist programmer, QA Test Specialist and Project Manager in my day job, I like where this is going.? And yes there is a but¡ ? But, I think that two if not three areas of communications are being combined in to this single topic: ¡°What will be the JMRI specification¡±.? These are JMRI, MQTT and Hardware and from what I read these are be talked about as if JMRI will set the spec for all three. ? Again my opinion so shout me down if you want but JMRI can only deal with how the MQTT Broker talks to JMRI and that may be the other way around as with DCC Systems.? (Each DCC Systems specs how JMRI will talk to it.)? How the MQTT Broker talks to the Hardware, I believe is already defined. ? JMRI has defined how scripts talk about Turnouts, Sensors, Signal Heads and Signal Mast as well as other items so putting this on a communications line to the MQTT is basically done. All that needs to be decided is the format to send to/read from ?MQTT, again what I read, that is mostly done as well /Trains/Turnout/Board ID/Item ID/Command or /Trains/Sensor/Board ID/Item ID/Data. ? How the MQTT Broker passes this information to the Hardware is up to the MQTT Broker designer/coder (the MQTT Specification).? As is how the Hardware gets the information from MQTT is already defined and how the Hardware deals with that information it gets it is up to the Hardware designer. ? There may well be two or more flavors of MQTT Brokers, as there are several flavors of DCC Systems. ? Just my two cents, feel free to disregard or flame as desired. ? Victor ? ? Victor Gilbert Supertendent Sugarwood Railroad ????????????? ___???????????? __ _????????????????????????????????????????????? _???????????? ___????????????? _?????? _??????????????????????????????? _?? ??????o O O? / __|?? _? _??? / _` |? __ _????? _ _? __ __ __? ___???? ___??? __| |?????????? | _ \?? __ _???? (_)???? | |????? _ _??? ___??? __ _??? __| |? ?????o?????? \__ \? | +| |?? \__, | / _` |??? | '_| \ V? V / / _ \?? / _ \? / _` |?????????? |?? /? / _` |??? | |???? | |???? | '_|? / _ \? / _` |? / _` |? ?????o?????? |___/?? \_,_|?? |___/? \__,_|?? _|_|_?? \_/\_/? \___/?? \___/? \__,_|?????????? |_|_\? \__,_|?? _|_|_?? _|_|_?? _|_|_?? \___/? \__,_|? \__,_|? ????TS__[O]? |"""""| |"""""| |"""""| |"""""| |"""""| |"""""| |"""""| |"""""| |"""""| |"""""| |"""""| |"""""| |"""""| |"""""| |"""""| |"""""| |"""""| |"""""| ??? {======|_|???? |_|???? |_|???? |_|???? |_|???? |_|???? |_|???? |_|???? |_|???? |_|???? |_|???? |_|???? |_|???? |_|???? |_|???? |_|???? |_|???? |_|???? | ? ./o--000'"`-0-0-'"`-0-0-'"`-0-0-'"`-0-0-'"`-0-0-'"`-0-0-'"`-0-0-'"`-0-0-'"`-0-0-'"`-0-0-'"`-0-0-'"`-0-0-'"`-0-0-'"`-0-0-'"`-0-0-'"`-0-0-'"`-0-0-'"`-0-0-'" ?
?
toggle quoted message
Show quoted text
From: [email protected] [mailto:[email protected]] On Behalf Of Speed
Sent: December-07-18 8:07 AM
To: [email protected]
Subject: Re: [jmriusers] MQTT Connection in JMRI ? A few more things to think about...
- will JMRI subscribe to every topic that contains a sensor/turnout/mast? I have found there are limits (in performance and memory) for subscribing to too many things. About a year ago I created this Jython script () to publish and subscribe to MQTT for "Virtual Masts" from within JMRI. I simply used the "Username" to keep the topic for each virtual mast, which was simple at the time: "mast.0003". Today I would go with /TxNamib/mast.0001/03, where the mast.0001 is the board where the masts are connected to.?
- Do we use retention? I found it convenient in my ESP implementation to save the state of the turnout in /TxNamib/servo.0003/01 as a '0' or a '1' with retention on, so when the ESP boots up, it knows where to move the servo without keeping the state saved in local EEPROM. I know some don't like physical addressing boards, but I found it convenient to simply subscribe to /TxNamib/servo.0003/# for all 16 turnouts connected to the ESP. So if retention is used, JMRI is going to turn on and get hundreds of messages (that it needs to keep up with) and also might have the "INCONSISTENT" state if it determines the turnout to be otherwise.?
- JMRI typically (or used to) only generate a message to hardware when there is a state change, so without retention in MQTT, the servo won't know where to go until there is a change in JMRI.
- Like Bob said, somethings work different than others. A block detector sensor is by nature controlled by the block detection circuit, but a global "I need the Dispatcher's attention" sensor could be turned on by more than one. (And who turns it off?). Masts should also be single sourced, in this case I would say JMRI determines the aspect.
- Will JMRI allow us to pick what messages to send? For example, "Active" and "Inactive" versus '1' and '0' versus 'True' or 'False'? Might not sound important here, but who wrote the code on the other side, it matters over there.
- And Masts having 'Clear', 'Approach' and Signal Heads using 'Green', 'Flashing Red', 'Dark', ...? Or do we start with only Heads?
- JMRI also need to allow, not only the username and password as mentioned earlier, but also the QOS (quality of service), and other (or future), settings.
[ Disclaimer about my Jython script: I can't promise the "threading" part is working well enough, sometimes you need to restart JMRI. The JButton on the Desktop was a simple way to remove the Listeners when you needed to rename a mast, instead of restarting JMRI. ]
|
Locked
Re: Double crossover problem in Panel Editor
Dave,
No..... but I did not do that the last time I was successful.? I tried it now and the first drag went OK but it would not let me drag the same turnout number to a second box. It said the "Duplicate Turnout Name assigned" and would not let me do it.
I am using a Kato Unitrack double crossover and all 4 turnouts operate with a single turnout number.? JRMI appears to be expecting 4 different turnout numbers to operate a double crossover. In addition, I don't understand how I was able to add the double turnout previously but not now.?
Paul D
|
Locked
Re: Double crossover problem in Panel Editor
Paul,
After you add the four turnouts using the "Add item to table", are you dragging the related entries to the ¡°undefined" boxes above the icons?
Dave Sand
toggle quoted message
Show quoted text
On Dec 7, 2018, at 10:36 AM, Paul Davidson <pdavidson@...> wrote:
Earlier I asked a question on forum about how to insert a double crossover in PE. After stumbling around, I finally got it to work. However, I no longer can get it to work (Ver 4.13.5). I must be doing something wrong. Here are the steps I am taking..
1) Open a new panel in PE
2) Use " Select the type of icon to add to panel" drop down to pick a "slip/3-way turnout/scissor"
3) In new window that opens, I select the button for "4 Turnouts"
4) I perform a "Add item to table" (using LT20 as an example)
5) At this point, I expect the "Add to Panel" button to become active so I can add the turnout to the panel but it will not become active
Somehow, this worked OK earlier with 4.13.5 but will not work now. The 5 steps work just fine for a left or right turnout but not for a double crossover (scissor on menu). I can go to a older panel I created when experimenting with adding a double crossover and the double crossover shows up and works fine. Driving me craze. What am I doing wrong?
Paul D
|
Locked
Re: MQTT Connection in JMRI
Does this mean that even the simplest HW device being directly addressed in an MQTT net needs a full TCP/IP Stack?
Not sure how that makes things simple.
Andy
---
This email has been checked for viruses by AVG.
|
Locked
Double crossover problem in Panel Editor
Earlier I asked a question on forum about how to insert a double crossover in PE. After stumbling around, I finally got it to work. However, I no longer can get it to work (Ver 4.13.5). I must be doing something wrong. Here are the steps I am taking..
1) Open a new panel in PE
2) Use " Select the type of icon to add to panel" drop down to pick a "slip/3-way turnout/scissor"
3) In new window that opens, I select the button for "4 Turnouts"
4) I perform a "Add item to table" (using LT20 as an example)
5) At this point, I expect the "Add to Panel" button to become active so I can add the turnout to the panel but it will not become active
Somehow, this worked OK earlier with 4.13.5 but will not work now. The 5 steps work just fine for a left or right turnout but not for a double crossover (scissor on menu).? I can go to a older panel I created when experimenting with adding a double crossover and the double crossover shows up and works fine. Driving me craze. What am I doing wrong?
Paul D
|
Locked
Re: MQTT Connection in JMRI
A few more things to think about...
- will JMRI subscribe to every topic that contains a sensor/turnout/mast? I have found there are limits (in performance and memory) for subscribing to too many things. About a year ago I created this Jython script () to publish and subscribe to MQTT for "Virtual Masts" from within JMRI. I simply used the "Username" to keep the topic for each virtual mast, which was simple at the time: "mast.0003". Today I would go with /TxNamib/mast.0001/03, where the mast.0001 is the board where the masts are connected to.?
- Do we use retention? I found it convenient in my ESP implementation to save the state of the turnout in /TxNamib/servo.0003/01 as a '0' or a '1' with retention on, so when the ESP boots up, it knows where to move the servo without keeping the state saved in local EEPROM. I know some don't like physical addressing boards, but I found it convenient to simply subscribe to /TxNamib/servo.0003/# for all 16 turnouts connected to the ESP. So if retention is used, JMRI is going to turn on and get hundreds of messages (that it needs to keep up with) and also might have the "INCONSISTENT" state if it determines the turnout to be otherwise.?
- JMRI typically (or used to) only generate a message to hardware when there is a state change, so without retention in MQTT, the servo won't know where to go until there is a change in JMRI.
- Like Bob said, somethings work different than others. A block detector sensor is by nature controlled by the block detection circuit, but a global "I need the Dispatcher's attention" sensor could be turned on by more than one. (And who turns it off?). Masts should also be single sourced, in this case I would say JMRI determines the aspect.
- Will JMRI allow us to pick what messages to send? For example, "Active" and "Inactive" versus '1' and '0' versus 'True' or 'False'? Might not sound important here, but who wrote the code on the other side, it matters over there.
- And Masts having 'Clear', 'Approach' and Signal Heads using 'Green', 'Flashing Red', 'Dark', ...? Or do we start with only Heads?
- JMRI also need to allow, not only the username and password as mentioned earlier, but also the QOS (quality of service), and other (or future), settings.
[ Disclaimer about my Jython script: I can't promise the "threading" part is working well enough, sometimes you need to restart JMRI. The JButton on the Desktop was a simple way to remove the Listeners when you needed to rename a mast, instead of restarting JMRI. ]
|
Locked
Re: MQTT Connection in JMRI
On Dec 7, 2018, at 09:35, Bob Jacobsen <rgj1927@...> wrote:
it would be great if the community could converge on how JMRI should work with MQTT devices.
Some boundary conditions:
1) JMRI has specific control metaphors: Turnout (i.e. one-bit digital output), Sensor (i.e. one-bit digital input), Light, SignalMast/SignalHead, Reporter, etc. They each function a bit differently. _How_ they work with MQTT is something that needs to be defined: What does it mean when a MQTT-based light wants to set intensity to 75%? What should a Sensor look for to decide whether it should go to ACTIVE, INACTIVE, UNKNOWN or INCONSISTENT, and if something else happens, what should it do? What do you mean by look for?
2) JMRI hardware flexibility comes from the system name metaphor. A Turnout (T) in the MQTT system (M) can have a system name like MT/stuff/morestuff/123/$5/dancing-bear-emoji for all JMRI cares. So you can put a lot of information into that system name to define the operation, so long as it can be unambiguously parsed. It¡¯s even possible to have different formats for the same underlying hardware devices, c.f. CT3001 vs CT3b1 and all the weird and wonderful MERGE CBUS formats.
Ideally it looks like this. /trains/hardwareID/item Hardware ID = Mac address Item = sub item on that device JRMI has a hardware id of MAC/item JSON in the payload = to the JSON used in WebSockets. This would make it THE most flexible, but the LEASE human readable.
So it would be _great_ if the MQTT community could come up with an addressing approach (which can have alternatives, i.e. with and without JSON) and an example of how a few cases, i.e. Sensor and Light. should work with it. We can then get that coded up for testing during the semester break that¡¯s coming.
Not sure who the MQTT community is¡but I can write use cases, if that is what you are asking.
Bob
--
Bob Jacobsen
rgj1927@...
|
Locked
Re: MQTT Connection in JMRI
it would be great if the community could converge on how JMRI should work with MQTT devices.
Some boundary conditions:
1) JMRI has specific control metaphors: Turnout (i.e. one-bit digital output), Sensor (i.e. one-bit digital input), Light, SignalMast/SignalHead, Reporter, etc. They each function a bit differently. _How_ they work with MQTT is something that needs to be defined: What does it mean when a MQTT-based light wants to set intensity to 75%? What should a Sensor look for to decide whether it should go to ACTIVE, INACTIVE, UNKNOWN or INCONSISTENT, and if something else happens, what should it do?
2) JMRI hardware flexibility comes from the system name metaphor. A Turnout (T) in the MQTT system (M) can have a system name like MT/stuff/morestuff/123/$5/dancing-bear-emoji for all JMRI cares. So you can put a lot of information into that system name to define the operation, so long as it can be unambiguously parsed. It¡¯s even possible to have different formats for the same underlying hardware devices, c.f. CT3001 vs CT3b1 and all the weird and wonderful MERGE CBUS formats.
So it would be _great_ if the MQTT community could come up with an addressing approach (which can have alternatives, i.e. with and without JSON) and an example of how a few cases, i.e. Sensor and Light. should work with it. We can then get that coded up for testing during the semester break that¡¯s coming.
Bob
--
Bob Jacobsen
rgj1927@...
|
Locked
Re: Blocks next to Crossovers are inaccessible as transit destinations
Lain.? Yes there are End Bumpers at every stop and each has a Signal Mast. Those Signal Masts are included in the SML and used for defining Sections.
Dave.? For your first question, I changed from automatic SML to manual SML because I was having issues running automatic SML with automatic trains.? With automatic SML, as soon as I created a train using Dispatcher, the train would start briefly, and then the signals would change to block the movement of the train.? This seems like a bug to me because as soon as I went to manual SML, it worked fine.? And as far as I can tell, the manual SML was identical to the automatic SML.? With automatic SML,?I would get an error on the console saying that the train was stopped because the next section was not allocated.? if I went into the SML logic at that point and updated it, it would clear the signal and the train would run automatically.? No clue.
Beyond that, I included the Signal Masts at those other locations so that I would have Signal Masts available to define Sections.? I guess it never occurred to me that you could have sections without signal masts.? I assumed that the sections were somehow tied into the Signal Mast Logic, probably because you can generate sections automatically, and they seem to run between signals.? ?Newbie mistake.? I'll take a look at that again and see what I can come up with. Given that, I also see now that those additional signals don't really provide value particularly if you use manual SML.? I was assuming the automatic tools under the Signal Mast Logic table would create a system for me that worked.? ?Perhaps they need a little work.? Thanks for the advice.
Thanks again,
Jack
|
Locked
Re: Using Digikeijs DR5000 with a Macintosh Computer
Thanks Bob
I tried the Ethernet connection but I have the same problem as with wifi. ?I can not use the z21app as the setting on the DR5000 are different. ?I will have have to figure out how to use the USB I guess. John
|
Locked
Re: MQTT Connection in JMRI
I agree with what everyone is saying. I do home automation also including node red. The key is to offer people flexibility, so they aren¡¯t forced to use other software if they choose not to. I guess like Nathan said you can use the WebSockets and create a WebSocket to MQTT gateway using nodered on a computer, or other custom methods such as an ESP8266/ESP32, or Raspberry Pi.?
I also do agree with the JSON too. JSON allows you to pass multiple pieces of information at once.?
I also agree that the Fast Clock then use a date also ?like Hans.
The key is to get information and data in and out in the shortest amount of time.
I think the key is to make JMRI flexible and powerful with IOT. This will allow a lot more control of not just the layout but the whole area where the layout is, such as room lighting, etc.
|
Locked
Re: MQTT Connection in JMRI
I am going to guess that the reason the path/topic is used normally is that it gives a hierarchy. So you can fo the # and + thing to segment what you are interested in. That said, almost every other MQTT device I have use JSON in the payload¡and node red loves a good JSON.
Standardizing all messages on JSON would allow a large degree of interoperability. Perhaps /trains/blocks/ then the normal JSON for blocks and so forth.
But the thing I keep coming back to is how do I make this as simple as I can on the sensor device so I can crank them out without writing a lot of code. I am exploring sending in the MAC+sub# and then using node red to map this to a JMRI sensor. So if you wanted to make this work without node red you¡¯d need to take the hardware field (or some UUID) on JMRI and map it to the third level in the topic: /trains/sensors/001422012345+1 with JSON of?{"type":"sensor","data":{"name":"001422012345+1¡±,"state":4}} or payload of 1 or 0 if people dislike the JSON.
I hope my thinking makes sense - the goal being I do not want to custom config my sensors, I just want to deploy them and virtually wire them up. I want the?Mac+sub because I wire up?multiple sensors to the same ESP and this?allows me to?separate out the input and output?really easily for bidirectional communications over MQTT.
Nathan
toggle quoted message
Show quoted text
On Dec 6, 2018, at 23:03, Dave < davemc@...> wrote:
Folks,
I'm also keen to see the MQTT implementation expanded to include sensors and signals (signal masts).
MQTT is a quite simple technology, easy to set up and use, no rocket science there - it's new to me but the learning curve is easy. As has been mentioned, having JMRI able to interface to the MQTT broker opens the way for people to easily expand on JMRI capabilities with different hardware and ¡°play¡±/explore new ideas.
The main issue at the moment seems to me to be agreement on how to structure the MQTT topics and payload, allowing flexibility for the downstream apps to perform the tasks required while remaining within the capabilities of a JMRI connection. That's where you need to tap into people's experiences to get a better idea what is preferred so you can begin with the end or multiple ends in mind. Obviously there has already been some consideration to come up with the /trains/track/turnout/1 structure.
Regards, Dave Mc.
|
Locked
Re: Mac and PC moving data
thanks Dave I got it work. I appreciate the help
Dan
|
Locked
Re: MQTT Connection in JMRI
Folks,
I'm also keen to see the MQTT implementation expanded to include sensors and signals (signal masts).
MQTT is a quite simple technology, easy to set up and use, no rocket science there - it's new to me but the learning curve is easy. As has been mentioned, having JMRI able to interface to the MQTT broker opens the way for people to easily expand on JMRI capabilities with different hardware and ¡°play¡±/explore new ideas.
The main issue at the moment seems to me to be agreement on how to structure the MQTT topics and payload, allowing flexibility for the downstream apps to perform the tasks required while remaining within the capabilities of a JMRI connection. That's where you need to tap into people's experiences to get a better idea what is preferred so you can begin with the end or multiple ends in mind. Obviously there has already been some consideration to come up with the /trains/track/turnout/1 structure.
Regards, Dave Mc.
|
Locked
Re: MQTT Connection in JMRI
So I have been pondering this: I use MQTT for everything, hence my interest with JMRI - I have a large saltwater aquarium and all my control equipment and monitoring is home made and sends signals around using MQTT, as does all my home automation stuff.
But as I thought about this¡I am not sure it really is needed for JMRI - I am trying to think where MQTT would be superior to WebSockets. Im using WebSockets in my iOS app, my physical throttle device (rotary encoder), and a host of other things.
Then I thought, ok, so I have a rasp pi zero for my MQTT broker and it also has node red on it and I can use the pins for input and output in node red too, so I made a flow that takes in JMRI type messages in either MQTT topics or JSON and changes both to JSON to send to the WebSockets on my JMRI computer.
The reason I wanted to put node red in there is I want an abstraction layer, I can take in GPIO from the zero, MQTT, and JSON and more¡and all end up in JMRI. I can even create a test harnesses to fake train data to experiment with my automation.
The real goal is to use either the macOS or AWS object tracker ML to send in positional data for all the open areas of the layout and then maybe (if I need them) ¡°normal¡± sensors for the hidden areas. A good camera above the layout it a hell of a lot fewer wires. Plus it turns out the tops of my locomotives are different enough I can even ID them from above. Working on stitching this all together now from prototype on a small track to a larger layout.
So yeah MQTT support: seems like a good thing to expand, as maybe people dont also want to fiddle with node red!
My vote is to keep the topics simple and use JSON payloads - they are already there in the WebSocket, so why not use the same ones?
Nathan
toggle quoted message
Show quoted text
On Dec 6, 2018, at 19:59, Hans < hans@...> wrote:
- I would suggest you make the topic user configurable. With fixed topics it starts to be real fun if more then one user uploads to a public server like HIveMQ ;-)
- In order to make everything simple I would suggest to make the topic structure as simple as possible, i.e. as few topics as possible. In general I find it easier to use one topic and embed the message e.g. in a JSON string that can be interpreted by the application
- Adding Fast Clock should be a no brainer, I think. Maybe you have seen my video on implementing the Fast Clock in Node RED and feeding it back to LocoNet via MQTT (). The advantage in case of the Digitrax system is that you now have an FC that also can support date, not just time.
Hans
|
Locked
Re: Mac and PC moving data
It would be in the Panel or Config file you saved, with the name of your choice.
It should be in User Files Location. If not, move it to there.
--
Dave in Australia
toggle quoted message
Show quoted text
On 7 Dec 2018, at 11:30 AM, dan@... wrote:
Thanks Dave. I think o have it except I can¡¯t find my turnout list?
|
I've been trying to create sections in readiness for transits and hopefully some automatic running, but while trying to create them, I am having problems with forward and reverse blocks either not appearing, or appearing twice, or even blocks are shown that are not physically (or logically) connected tothe section that I am trying to create. On occasions a reverse block is not shown, but that block shows the forward link in its own section. For example?
Physical Link: Goods line - Yard Approach - Down Loop - Spur / Station Approach
Section Down Loop ;Yard Approach - Down Loop (no follow on)
Section Loop Spur? ? ? ? ? ? ? ? ? ? ? ? ? ?Down Loop - Spur
I've Added some screen shots to folder swisschris Sections and added a JMRI to a file folder (swissChris Sections)
Also the 2 red crosses are signal masts that i added in a previous session.reloaded it only to find that the masts and SML had been deleted (but not the heads that i had created). Any ideas please?
JMRI PanelPro 4.12
Win 10
Gaugemaster (MRC) Prodigy
|
Locked
Re: MQTT Connection in JMRI
- I would suggest you make the topic user configurable. With fixed topics it starts to be real fun if more then one user uploads to a public server like HIveMQ ;-)
- In order to make everything simple I would suggest to make the topic structure as simple as possible, i.e. as few topics as possible. In general I find it easier to use one topic and embed the message e.g. in a JSON string that can be interpreted by the application
- Adding Fast Clock should be a no brainer, I think. Maybe you have seen my video on implementing the Fast Clock in Node RED and feeding it back to LocoNet via MQTT (https://www.youtube.com/watch?v=6RjzpHJu-vU). The advantage in case of the Digitrax system is that you now have an FC that also can support date, not just time.
Hans
|
Locked
Re: Mac and PC moving data
Thanks Dave. I think o have it except I can¡¯t find my turnout list?
dan
|
Locked
Re: MQTT Connection in JMRI
Bob,? I'd be happy to help getting MQTT subscriptions on the inside of JMRI. We already have it working in the Python side, but built-in would be better.?
I have about 25 ESP-8266 ESP-01 modules up and running (with Paho MQTT in JavaScript (mqttws31.js) )? already, so testing it is also easy.
Hope I can help!
Speed
(aka Gert 'Jim' Muller)
|
Paul Davidson
Bob Jacobsen
Jack Basiago
John Kabesh
SwissChris