[sr-dev] [kamailio/kamailio] sequence breaking DMQ bus (#1349)

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

[sr-dev] [kamailio/kamailio] sequence breaking DMQ bus (#1349)

Daniel-Constantin Mierla-3

I reproduced in a lab with 3 Kamailio servers

kamailio_first  127.0.0.101
modparam("dmq", "server_address", "sip:127.0.0.101:5060")
modparam("dmq", "notification_address", "sip:127.0.0.103:5060")
modparam("dmq", "multi_notify", 1)
modparam("dmq", "num_workers", 2)
modparam("dmq", "ping_interval", 15)

kamailio_second 127.0.0.102 notification_peer: 127.0.0.103
modparam("dmq", "server_address", "sip:127.0.0.102:5060")
modparam("dmq", "notification_address", "sip:127.0.0.103:5060")
modparam("dmq", "multi_notify", 1)
modparam("dmq", "num_workers", 2)
modparam("dmq", "ping_interval", 15)

kamailio_third  127.0.0.103 notification_peer: 127.0.0.101
modparam("dmq", "server_address", "sip:127.0.0.103:5060")
modparam("dmq", "notification_address", "sip:127.0.0.101:5060")
modparam("dmq", "multi_notify", 1)
modparam("dmq", "num_workers", 2)
modparam("dmq", "ping_interval", 15)

kamcmd -s /var/run/kamailio_$1/kamailio_ctl dmq.list_nodes | grep -e host -e status -e last -e local | tr '\n' ' ' | sed -e 's/host/\n&/g'

show dmq bus: first	
host: 127.0.0.102 	status: 2 	last_notification: 0 	local: 0 	
host: 127.0.0.103 	status: 2 	last_notification: 0 	local: 0 	
host: 127.0.0.101 	status: 2 	last_notification: 0 	local: 1 
show dmq bus: second	
host: 127.0.0.101 	status: 2 	last_notification: 0 	local: 0 	
host: 127.0.0.103 	status: 2 	last_notification: 0 	local: 0 	
host: 127.0.0.102 	status: 2 	last_notification: 0 	local: 1 
show dmq bus: third	
host: 127.0.0.102 	status: 2 	last_notification: 0 	local: 0 	
host: 127.0.0.101 	status: 2 	last_notification: 0 	local: 0 	
host: 127.0.0.103 	status: 2 	last_notification: 0 	local: 1 
/etc/init.d/kamailio_first stop
/etc/init.d/kamailio_third stop

/etc/init.d/kamailio_third start
/etc/init.d/kamailio_first start

kamcmd -s /var/run/kamailio_$1/kamailio_ctl dmq.list_nodes | grep -e host -e status -e last -e local | tr '\n' ' ' | sed -e 's/host/\n&/g'

show dmq bus: first	
host: 127.0.0.103 	status: 2 	last_notification: 0 	local: 0 	
host: 127.0.0.101 	status: 2 	last_notification: 0 	local: 1 
show dmq bus: second	
host: 127.0.0.101 	status: 8 	last_notification: 0 	local: 0 	
host: 127.0.0.103 	status: 8 	last_notification: 0 	local: 0 	
host: 127.0.0.102 	status: 2 	last_notification: 0 	local: 1 
show dmq bus: third	
host: 127.0.0.101 	status: 2 	last_notification: 0 	local: 0 	
host: 127.0.0.103 	status: 2 	last_notification: 0 	local: 1 

The bus will remain broken, I think there is a second scenario that can break.

The notes I took when I found this (I consider this was a limitation not a bug)
Why : when Kamailio_third is shutting down, it will send a message to every peers telling them he is now inactive so they will not try to contact him again.
once restarted no nodes will contact him because they have an inactive state for him, at this point the only way he can learn about the other nodes is by contacting Kamailio_first, however Kamailio_first will not know about the other nodes.


You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.

<script type="application/json" data-scope="inboxmarkup">{"api_version":"1.0","publisher":{"api_key":"05dde50f1d1a384dd78767c55493e4bb","name":"GitHub"},"entity":{"external_key":"github/kamailio/kamailio","title":"kamailio/kamailio","subtitle":"GitHub repository","main_image_url":"https://cloud.githubusercontent.com/assets/143418/17495839/a5054eac-5d88-11e6-95fc-7290892c7bb5.png","avatar_image_url":"https://cloud.githubusercontent.com/assets/143418/15842166/7c72db34-2c0b-11e6-9aed-b52498112777.png","action":{"name":"Open in GitHub","url":"https://github.com/kamailio/kamailio"}},"updates":{"snippets":[{"icon":"DESCRIPTION","message":"sequence breaking DMQ bus (#1349)"}],"action":{"name":"View Issue","url":"https://github.com/kamailio/kamailio/issues/1349"}}}</script>
_______________________________________________
Kamailio (SER) - Development Mailing List
[hidden email]
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
Reply | Threaded
Open this post in threaded view
|

Re: [kamailio/kamailio] sequence breaking DMQ bus (#1349)

Daniel-Constantin Mierla-3

Hi Charles, I guess we can fix this, I think there is a second case where things can break I will try to find it a document it as well.
Or maybe I should simply use different config ...


You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.

<script type="application/json" data-scope="inboxmarkup">{"api_version":"1.0","publisher":{"api_key":"05dde50f1d1a384dd78767c55493e4bb","name":"GitHub"},"entity":{"external_key":"github/kamailio/kamailio","title":"kamailio/kamailio","subtitle":"GitHub repository","main_image_url":"https://cloud.githubusercontent.com/assets/143418/17495839/a5054eac-5d88-11e6-95fc-7290892c7bb5.png","avatar_image_url":"https://cloud.githubusercontent.com/assets/143418/15842166/7c72db34-2c0b-11e6-9aed-b52498112777.png","action":{"name":"Open in GitHub","url":"https://github.com/kamailio/kamailio"}},"updates":{"snippets":[{"icon":"PERSON","message":"@jchavanton in #1349: Hi Charles, I guess we can fix this, I think there is a second case where things can break I will try to find it a document it as well.\r\nOr maybe I should simply use different config ..."}],"action":{"name":"View Issue","url":"https://github.com/kamailio/kamailio/issues/1349#issuecomment-348730398"}}}</script>
_______________________________________________
Kamailio (SER) - Development Mailing List
[hidden email]
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
Reply | Threaded
Open this post in threaded view
|

Re: [kamailio/kamailio] sequence breaking DMQ bus (#1349)

Daniel-Constantin Mierla-3
In reply to this post by Daniel-Constantin Mierla-3

I also tried the ring strategy but, I did not like the fact that if you loose one server the ring may get broken.

Maybe this is still better in the end but when you have a lot of server it is more likely that one server may be removed from service.

This is why I am testing with a "master" node
Not sure what is you opinion

Example of broken state with a ring config and 4 servers:

/etc/init.d/kamailio_second stop
/etc/init.d/kamailio_third stop
/etc/init.d/kamailio_second start

You end up with

show dmq bus: first	
host: 127.0.0.103 	status: 8 	last_notification: 0 	local: 0 	
host: 127.0.0.104 	status: 2 	last_notification: 0 	local: 0 	
host: 127.0.0.102 	status: 8 	last_notification: 0 	local: 0 	
host: 127.0.0.101 	status: 2 	last_notification: 0 	local: 1 
show dmq bus: second	
host: 127.0.0.103 	status: 2 	last_notification: 0 	local: 0 	
host: 127.0.0.102 	status: 2 	last_notification: 0 	local: 1 
show dmq bus: third          not running !
show dmq bus: fourth	
host: 127.0.0.102 	status: 8 	last_notification: 0 	local: 0 	
host: 127.0.0.103 	status: 8 	last_notification: 0 	local: 0 	
host: 127.0.0.101 	status: 2 	last_notification: 0 	local: 0 	
host: 127.0.0.104 	status: 2 	last_notification: 0 	local: 1 


You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.

<script type="application/json" data-scope="inboxmarkup">{"api_version":"1.0","publisher":{"api_key":"05dde50f1d1a384dd78767c55493e4bb","name":"GitHub"},"entity":{"external_key":"github/kamailio/kamailio","title":"kamailio/kamailio","subtitle":"GitHub repository","main_image_url":"https://cloud.githubusercontent.com/assets/143418/17495839/a5054eac-5d88-11e6-95fc-7290892c7bb5.png","avatar_image_url":"https://cloud.githubusercontent.com/assets/143418/15842166/7c72db34-2c0b-11e6-9aed-b52498112777.png","action":{"name":"Open in GitHub","url":"https://github.com/kamailio/kamailio"}},"updates":{"snippets":[{"icon":"PERSON","message":"@jchavanton in #1349: I also tried the ring strategy but, I did not like the fact that if you loose one server the ring may get broken. \r\n\r\nMaybe this is still better in the end but when you have a lot of server it is more likely that one server may be removed from service.\r\n\r\nThis is why I am testing with a \"master\" node\r\nNot sure what is you opinion \r\n\r\nExample of broken state with a ring config and 4 servers: \r\n```\r\n/etc/init.d/kamailio_second stop\r\n/etc/init.d/kamailio_third stop\r\n/etc/init.d/kamailio_second start\r\n```\r\n\r\nYou end up with \r\n```\r\nshow dmq bus: first\t\r\nhost: 127.0.0.103 \tstatus: 8 \tlast_notification: 0 \tlocal: 0 \t\r\nhost: 127.0.0.104 \tstatus: 2 \tlast_notification: 0 \tlocal: 0 \t\r\nhost: 127.0.0.102 \tstatus: 8 \tlast_notification: 0 \tlocal: 0 \t\r\nhost: 127.0.0.101 \tstatus: 2 \tlast_notification: 0 \tlocal: 1 \r\nshow dmq bus: second\t\r\nhost: 127.0.0.103 \tstatus: 2 \tlast_notification: 0 \tlocal: 0 \t\r\nhost: 127.0.0.102 \tstatus: 2 \tlast_notification: 0 \tlocal: 1 \r\nshow dmq bus: third not running !\r\nshow dmq bus: fourth\t\r\nhost: 127.0.0.102 \tstatus: 8 \tlast_notification: 0 \tlocal: 0 \t\r\nhost: 127.0.0.103 \tstatus: 8 \tlast_notification: 0 \tlocal: 0 \t\r\nhost: 127.0.0.101 \tstatus: 2 \tlast_notification: 0 \tlocal: 0 \t\r\nhost: 127.0.0.104 \tstatus: 2 \tlast_notification: 0 \tlocal: 1 \r\n```\r\n"}],"action":{"name":"View Issue","url":"https://github.com/kamailio/kamailio/issues/1349#issuecomment-348732404"}}}</script>
_______________________________________________
Kamailio (SER) - Development Mailing List
[hidden email]
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
Reply | Threaded
Open this post in threaded view
|

Re: [kamailio/kamailio] sequence breaking DMQ bus (#1349)

Daniel-Constantin Mierla-3
In reply to this post by Daniel-Constantin Mierla-3

Thanks for reporting and for taking the time to test different scenarios.

Using your examples I will try to reproduce locally but out of interest, do you observe the same behaviour if notification address is an FQDN resolving to multiple IPs?


You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.

<script type="application/json" data-scope="inboxmarkup">{"api_version":"1.0","publisher":{"api_key":"05dde50f1d1a384dd78767c55493e4bb","name":"GitHub"},"entity":{"external_key":"github/kamailio/kamailio","title":"kamailio/kamailio","subtitle":"GitHub repository","main_image_url":"https://cloud.githubusercontent.com/assets/143418/17495839/a5054eac-5d88-11e6-95fc-7290892c7bb5.png","avatar_image_url":"https://cloud.githubusercontent.com/assets/143418/15842166/7c72db34-2c0b-11e6-9aed-b52498112777.png","action":{"name":"Open in GitHub","url":"https://github.com/kamailio/kamailio"}},"updates":{"snippets":[{"icon":"PERSON","message":"@charlesrchance in #1349: Thanks for reporting and for taking the time to test different scenarios.\r\n\r\nUsing your examples I will try to reproduce locally but out of interest, do you observe the same behaviour if notification address is an FQDN resolving to multiple IPs?"}],"action":{"name":"View Issue","url":"https://github.com/kamailio/kamailio/issues/1349#issuecomment-349255804"}}}</script>
_______________________________________________
Kamailio (SER) - Development Mailing List
[hidden email]
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
Reply | Threaded
Open this post in threaded view
|

Re: [kamailio/kamailio] sequence breaking DMQ bus (#1349)

Daniel-Constantin Mierla-3
In reply to this post by Daniel-Constantin Mierla-3

I am guessing FQDN resolving to multiple IPs will greatly improve things !
I will test it and maybe we could document a wiki page on how to configure DMQ bus


You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.

<script type="application/json" data-scope="inboxmarkup">{"api_version":"1.0","publisher":{"api_key":"05dde50f1d1a384dd78767c55493e4bb","name":"GitHub"},"entity":{"external_key":"github/kamailio/kamailio","title":"kamailio/kamailio","subtitle":"GitHub repository","main_image_url":"https://cloud.githubusercontent.com/assets/143418/17495839/a5054eac-5d88-11e6-95fc-7290892c7bb5.png","avatar_image_url":"https://cloud.githubusercontent.com/assets/143418/15842166/7c72db34-2c0b-11e6-9aed-b52498112777.png","action":{"name":"Open in GitHub","url":"https://github.com/kamailio/kamailio"}},"updates":{"snippets":[{"icon":"PERSON","message":"@jchavanton in #1349: I am guessing FQDN resolving to multiple IPs will greatly improve things !\r\nI will test it and maybe we could document a wiki page on how to configure DMQ bus"}],"action":{"name":"View Issue","url":"https://github.com/kamailio/kamailio/issues/1349#issuecomment-349711920"}}}</script>
_______________________________________________
Kamailio (SER) - Development Mailing List
[hidden email]
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
Reply | Threaded
Open this post in threaded view
|

Re: [kamailio/kamailio] sequence breaking DMQ bus (#1349)

Daniel-Constantin Mierla-3
In reply to this post by Daniel-Constantin Mierla-3

@jchavanton - would you mind running some tests again with 851e610 to see if there is an improvement?


You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.

<script type="application/json" data-scope="inboxmarkup">{"api_version":"1.0","publisher":{"api_key":"05dde50f1d1a384dd78767c55493e4bb","name":"GitHub"},"entity":{"external_key":"github/kamailio/kamailio","title":"kamailio/kamailio","subtitle":"GitHub repository","main_image_url":"https://cloud.githubusercontent.com/assets/143418/17495839/a5054eac-5d88-11e6-95fc-7290892c7bb5.png","avatar_image_url":"https://cloud.githubusercontent.com/assets/143418/15842166/7c72db34-2c0b-11e6-9aed-b52498112777.png","action":{"name":"Open in GitHub","url":"https://github.com/kamailio/kamailio"}},"updates":{"snippets":[{"icon":"PERSON","message":"@charlesrchance in #1349: @jchavanton - would you mind running some tests again with 851e610 to see if there is an improvement?"}],"action":{"name":"View Issue","url":"https://github.com/kamailio/kamailio/issues/1349#issuecomment-353100886"}}}</script>
_______________________________________________
Kamailio (SER) - Development Mailing List
[hidden email]
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
Reply | Threaded
Open this post in threaded view
|

Re: [kamailio/kamailio] sequence breaking DMQ bus (#1349)

Daniel-Constantin Mierla-3
In reply to this post by Daniel-Constantin Mierla-3

Hi, I am still thinking about finding some time to test this in a few days / weeks ...


You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.

<script type="application/json" data-scope="inboxmarkup">{"api_version":"1.0","publisher":{"api_key":"05dde50f1d1a384dd78767c55493e4bb","name":"GitHub"},"entity":{"external_key":"github/kamailio/kamailio","title":"kamailio/kamailio","subtitle":"GitHub repository","main_image_url":"https://cloud.githubusercontent.com/assets/143418/17495839/a5054eac-5d88-11e6-95fc-7290892c7bb5.png","avatar_image_url":"https://cloud.githubusercontent.com/assets/143418/15842166/7c72db34-2c0b-11e6-9aed-b52498112777.png","action":{"name":"Open in GitHub","url":"https://github.com/kamailio/kamailio"}},"updates":{"snippets":[{"icon":"PERSON","message":"@jchavanton in #1349: Hi, I am still thinking about finding some time to test this in a few days / weeks ..."}],"action":{"name":"View Issue","url":"https://github.com/kamailio/kamailio/issues/1349#issuecomment-356127602"}}}</script>
_______________________________________________
Kamailio (SER) - Development Mailing List
[hidden email]
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
Reply | Threaded
Open this post in threaded view
|

Re: [kamailio/kamailio] sequence breaking DMQ bus (#1349)

Daniel-Constantin Mierla-3
In reply to this post by Daniel-Constantin Mierla-3

Closing this one, following the patch referenced above. If still an issue, reopen or create a new issue as the module got new code inside it.


You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.

<script type="application/json" data-scope="inboxmarkup">{"api_version":"1.0","publisher":{"api_key":"05dde50f1d1a384dd78767c55493e4bb","name":"GitHub"},"entity":{"external_key":"github/kamailio/kamailio","title":"kamailio/kamailio","subtitle":"GitHub repository","main_image_url":"https://cloud.githubusercontent.com/assets/143418/17495839/a5054eac-5d88-11e6-95fc-7290892c7bb5.png","avatar_image_url":"https://cloud.githubusercontent.com/assets/143418/15842166/7c72db34-2c0b-11e6-9aed-b52498112777.png","action":{"name":"Open in GitHub","url":"https://github.com/kamailio/kamailio"}},"updates":{"snippets":[{"icon":"PERSON","message":"@miconda in #1349: Closing this one, following the patch referenced above. If still an issue, reopen or create a new issue as the module got new code inside it."}],"action":{"name":"View Issue","url":"https://github.com/kamailio/kamailio/issues/1349#issuecomment-366169243"}}}</script>
_______________________________________________
Kamailio (SER) - Development Mailing List
[hidden email]
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
Reply | Threaded
Open this post in threaded view
|

Re: [kamailio/kamailio] sequence breaking DMQ bus (#1349)

Daniel-Constantin Mierla-3
In reply to this post by Daniel-Constantin Mierla-3

Closed #1349.


You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.

<script type="application/json" data-scope="inboxmarkup">{"api_version":"1.0","publisher":{"api_key":"05dde50f1d1a384dd78767c55493e4bb","name":"GitHub"},"entity":{"external_key":"github/kamailio/kamailio","title":"kamailio/kamailio","subtitle":"GitHub repository","main_image_url":"https://cloud.githubusercontent.com/assets/143418/17495839/a5054eac-5d88-11e6-95fc-7290892c7bb5.png","avatar_image_url":"https://cloud.githubusercontent.com/assets/143418/15842166/7c72db34-2c0b-11e6-9aed-b52498112777.png","action":{"name":"Open in GitHub","url":"https://github.com/kamailio/kamailio"}},"updates":{"snippets":[{"icon":"DESCRIPTION","message":"Closed #1349."}],"action":{"name":"View Issue","url":"https://github.com/kamailio/kamailio/issues/1349#event-1477417229"}}}</script>
_______________________________________________
Kamailio (SER) - Development Mailing List
[hidden email]
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev