[sr-dev] Kamailio 5.1 nightly (rc2) crashing - udp_rcv_loop(): print buffer building failed

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

[sr-dev] Kamailio 5.1 nightly (rc2) crashing - udp_rcv_loop(): print buffer building failed

George Diamantopoulos
Hello,

I've just installed kamailio 5.1 nightly from the debian repo for stretch. I didn't change anything in the configuration compared to 5.0.4 (which I was testing previously), apart from the database structure as found in the wiki for upgrading from 5.0.x.

Now kamailio crashes after a while, with the following output in the logs:

Nov 29 15:47:05 modcom-sbc-dev1 kamailio[2497]:  3(2502) ERROR: <core> [core/udp_server.c:481]: udp_rcv_loop(): print buffer building failed
Nov 29 15:47:05 modcom-sbc-dev1 kamailio[2497]: 121(2620) CRITICAL: <core> [core/pass_fd.c:277]: receive_fd(): EOF on 26
Nov 29 15:47:05 modcom-sbc-dev1 kamailio[2497]: 121(2620) DEBUG: <core> [core/tcp_main.c:3512]: handle_ser_child(): dead child 3, pid 2502 (shutting down?)
Nov 29 15:47:05 modcom-sbc-dev1 kamailio[2497]: 121(2620) DEBUG: <core> [core/io_wait.h:602]: io_watch_del(): DBG: io_watch_del (0x55ae7132ffc0, 26, -1, 0x0) fd_no=126 called
Nov 29 15:47:05 modcom-sbc-dev1 kamailio[2497]:  0(2499) ALERT: <core> [main.c:741]: handle_sigs(): child process 2502 exited normally, status=255
Nov 29 15:47:05 modcom-sbc-dev1 kamailio[2497]:  0(2499) INFO: <core> [main.c:771]: handle_sigs(): terminating due to SIGCHLD
Nov 29 15:47:05 modcom-sbc-dev1 kamailio[2497]:  0(2499) DEBUG: <core> [main.c:773]: handle_sigs(): terminating due to SIGCHLD
Nov 29 15:47:05 modcom-sbc-dev1 kamailio[2497]: 118(2617) INFO: <core> [main.c:826]: sig_usr(): signal 15 received

Let me know if I can somehow help dissect this issue, I don't really know where to start. Thanks.

BR,
George

_______________________________________________
Kamailio (SER) - Development Mailing List
[hidden email]
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
Reply | Threaded
Open this post in threaded view
|

Re: Kamailio 5.1 nightly (rc2) crashing - udp_rcv_loop(): print buffer building failed

George Diamantopoulos
I've been trying to get a core dump on this for hours now, but I just can't get one! Is there anything special needed with this build on debian stretch? Checked ulimits, disable_core_dump, /etc/default/kamailio, started it as root (-u root -g root), from systemd with user kamailio, with working directory set in command line and in config file, I just can't get a core dump...

Thanks

On 29 November 2017 at 18:10, George Diamantopoulos <[hidden email]> wrote:
Hello,

I've just installed kamailio 5.1 nightly from the debian repo for stretch. I didn't change anything in the configuration compared to 5.0.4 (which I was testing previously), apart from the database structure as found in the wiki for upgrading from 5.0.x.

Now kamailio crashes after a while, with the following output in the logs:

Nov 29 15:47:05 modcom-sbc-dev1 kamailio[2497]:  3(2502) ERROR: <core> [core/udp_server.c:481]: udp_rcv_loop(): print buffer building failed
Nov 29 15:47:05 modcom-sbc-dev1 kamailio[2497]: 121(2620) CRITICAL: <core> [core/pass_fd.c:277]: receive_fd(): EOF on 26
Nov 29 15:47:05 modcom-sbc-dev1 kamailio[2497]: 121(2620) DEBUG: <core> [core/tcp_main.c:3512]: handle_ser_child(): dead child 3, pid 2502 (shutting down?)
Nov 29 15:47:05 modcom-sbc-dev1 kamailio[2497]: 121(2620) DEBUG: <core> [core/io_wait.h:602]: io_watch_del(): DBG: io_watch_del (0x55ae7132ffc0, 26, -1, 0x0) fd_no=126 called
Nov 29 15:47:05 modcom-sbc-dev1 kamailio[2497]:  0(2499) ALERT: <core> [main.c:741]: handle_sigs(): child process 2502 exited normally, status=255
Nov 29 15:47:05 modcom-sbc-dev1 kamailio[2497]:  0(2499) INFO: <core> [main.c:771]: handle_sigs(): terminating due to SIGCHLD
Nov 29 15:47:05 modcom-sbc-dev1 kamailio[2497]:  0(2499) DEBUG: <core> [main.c:773]: handle_sigs(): terminating due to SIGCHLD
Nov 29 15:47:05 modcom-sbc-dev1 kamailio[2497]: 118(2617) INFO: <core> [main.c:826]: sig_usr(): signal 15 received

Let me know if I can somehow help dissect this issue, I don't really know where to start. Thanks.

BR,
George


_______________________________________________
Kamailio (SER) - Development Mailing List
[hidden email]
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
Reply | Threaded
Open this post in threaded view
|

Re: Kamailio 5.1 nightly (rc2) crashing - udp_rcv_loop(): print buffer building failed

George Diamantopoulos
Oh, I also tried with 5.2.0-dev1 nightly and it also crashes in the same way...

On 30 November 2017 at 01:07, George Diamantopoulos <[hidden email]> wrote:
I've been trying to get a core dump on this for hours now, but I just can't get one! Is there anything special needed with this build on debian stretch? Checked ulimits, disable_core_dump, /etc/default/kamailio, started it as root (-u root -g root), from systemd with user kamailio, with working directory set in command line and in config file, I just can't get a core dump...

Thanks

On 29 November 2017 at 18:10, George Diamantopoulos <[hidden email]> wrote:
Hello,

I've just installed kamailio 5.1 nightly from the debian repo for stretch. I didn't change anything in the configuration compared to 5.0.4 (which I was testing previously), apart from the database structure as found in the wiki for upgrading from 5.0.x.

Now kamailio crashes after a while, with the following output in the logs:

Nov 29 15:47:05 modcom-sbc-dev1 kamailio[2497]:  3(2502) ERROR: <core> [core/udp_server.c:481]: udp_rcv_loop(): print buffer building failed
Nov 29 15:47:05 modcom-sbc-dev1 kamailio[2497]: 121(2620) CRITICAL: <core> [core/pass_fd.c:277]: receive_fd(): EOF on 26
Nov 29 15:47:05 modcom-sbc-dev1 kamailio[2497]: 121(2620) DEBUG: <core> [core/tcp_main.c:3512]: handle_ser_child(): dead child 3, pid 2502 (shutting down?)
Nov 29 15:47:05 modcom-sbc-dev1 kamailio[2497]: 121(2620) DEBUG: <core> [core/io_wait.h:602]: io_watch_del(): DBG: io_watch_del (0x55ae7132ffc0, 26, -1, 0x0) fd_no=126 called
Nov 29 15:47:05 modcom-sbc-dev1 kamailio[2497]:  0(2499) ALERT: <core> [main.c:741]: handle_sigs(): child process 2502 exited normally, status=255
Nov 29 15:47:05 modcom-sbc-dev1 kamailio[2497]:  0(2499) INFO: <core> [main.c:771]: handle_sigs(): terminating due to SIGCHLD
Nov 29 15:47:05 modcom-sbc-dev1 kamailio[2497]:  0(2499) DEBUG: <core> [main.c:773]: handle_sigs(): terminating due to SIGCHLD
Nov 29 15:47:05 modcom-sbc-dev1 kamailio[2497]: 118(2617) INFO: <core> [main.c:826]: sig_usr(): signal 15 received

Let me know if I can somehow help dissect this issue, I don't really know where to start. Thanks.

BR,
George



_______________________________________________
Kamailio (SER) - Development Mailing List
[hidden email]
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
Reply | Threaded
Open this post in threaded view
|

Re: Kamailio 5.1 nightly (rc2) crashing - udp_rcv_loop(): print buffer building failed

George Diamantopoulos
OK, so I performed a clean installation on a new debian stretch instance I haven't been able to reproduce the issue so far. Either something changed in the latest nightly, or the fact that I upgraded from 5.0.4 was causing the issue.

The deployment is automated with ansible, so there shouldn't be any deviations in the configuration of the host. Only other difference (apart from the clean kamailio install) should be rtpengine's building, which was performed anew. However, I don't see how that might have affected things (after all, I don't believe kamailio is a build time dependency for rtpengine)...

On 30 November 2017 at 01:08, George Diamantopoulos <[hidden email]> wrote:
Oh, I also tried with 5.2.0-dev1 nightly and it also crashes in the same way...

On 30 November 2017 at 01:07, George Diamantopoulos <[hidden email]> wrote:
I've been trying to get a core dump on this for hours now, but I just can't get one! Is there anything special needed with this build on debian stretch? Checked ulimits, disable_core_dump, /etc/default/kamailio, started it as root (-u root -g root), from systemd with user kamailio, with working directory set in command line and in config file, I just can't get a core dump...

Thanks

On 29 November 2017 at 18:10, George Diamantopoulos <[hidden email]> wrote:
Hello,

I've just installed kamailio 5.1 nightly from the debian repo for stretch. I didn't change anything in the configuration compared to 5.0.4 (which I was testing previously), apart from the database structure as found in the wiki for upgrading from 5.0.x.

Now kamailio crashes after a while, with the following output in the logs:

Nov 29 15:47:05 modcom-sbc-dev1 kamailio[2497]:  3(2502) ERROR: <core> [core/udp_server.c:481]: udp_rcv_loop(): print buffer building failed
Nov 29 15:47:05 modcom-sbc-dev1 kamailio[2497]: 121(2620) CRITICAL: <core> [core/pass_fd.c:277]: receive_fd(): EOF on 26
Nov 29 15:47:05 modcom-sbc-dev1 kamailio[2497]: 121(2620) DEBUG: <core> [core/tcp_main.c:3512]: handle_ser_child(): dead child 3, pid 2502 (shutting down?)
Nov 29 15:47:05 modcom-sbc-dev1 kamailio[2497]: 121(2620) DEBUG: <core> [core/io_wait.h:602]: io_watch_del(): DBG: io_watch_del (0x55ae7132ffc0, 26, -1, 0x0) fd_no=126 called
Nov 29 15:47:05 modcom-sbc-dev1 kamailio[2497]:  0(2499) ALERT: <core> [main.c:741]: handle_sigs(): child process 2502 exited normally, status=255
Nov 29 15:47:05 modcom-sbc-dev1 kamailio[2497]:  0(2499) INFO: <core> [main.c:771]: handle_sigs(): terminating due to SIGCHLD
Nov 29 15:47:05 modcom-sbc-dev1 kamailio[2497]:  0(2499) DEBUG: <core> [main.c:773]: handle_sigs(): terminating due to SIGCHLD
Nov 29 15:47:05 modcom-sbc-dev1 kamailio[2497]: 118(2617) INFO: <core> [main.c:826]: sig_usr(): signal 15 received

Let me know if I can somehow help dissect this issue, I don't really know where to start. Thanks.

BR,
George




_______________________________________________
Kamailio (SER) - Development Mailing List
[hidden email]
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
Reply | Threaded
Open this post in threaded view
|

Re: Kamailio 5.1 nightly (rc2) crashing - udp_rcv_loop(): print buffer building failed

Daniel-Constantin Mierla

Hello,

did you run with debug=3 in kamailio.cfg? That piece of code is new in 5.1, but executed only when debug is at least 3 (which is for troubleshooting purposes, not for production). Anyhow, I will check a bit that part to see if anything is wrong there...

Cheers,
Daniel


On 30.11.17 18:52, George Diamantopoulos wrote:
OK, so I performed a clean installation on a new debian stretch instance I haven't been able to reproduce the issue so far. Either something changed in the latest nightly, or the fact that I upgraded from 5.0.4 was causing the issue.

The deployment is automated with ansible, so there shouldn't be any deviations in the configuration of the host. Only other difference (apart from the clean kamailio install) should be rtpengine's building, which was performed anew. However, I don't see how that might have affected things (after all, I don't believe kamailio is a build time dependency for rtpengine)...

On 30 November 2017 at 01:08, George Diamantopoulos <[hidden email]> wrote:
Oh, I also tried with 5.2.0-dev1 nightly and it also crashes in the same way...

On 30 November 2017 at 01:07, George Diamantopoulos <[hidden email]> wrote:
I've been trying to get a core dump on this for hours now, but I just can't get one! Is there anything special needed with this build on debian stretch? Checked ulimits, disable_core_dump, /etc/default/kamailio, started it as root (-u root -g root), from systemd with user kamailio, with working directory set in command line and in config file, I just can't get a core dump...

Thanks

On 29 November 2017 at 18:10, George Diamantopoulos <[hidden email]> wrote:
Hello,

I've just installed kamailio 5.1 nightly from the debian repo for stretch. I didn't change anything in the configuration compared to 5.0.4 (which I was testing previously), apart from the database structure as found in the wiki for upgrading from 5.0.x.

Now kamailio crashes after a while, with the following output in the logs:

Nov 29 15:47:05 modcom-sbc-dev1 kamailio[2497]:  3(2502) ERROR: <core> [core/udp_server.c:481]: udp_rcv_loop(): print buffer building failed
Nov 29 15:47:05 modcom-sbc-dev1 kamailio[2497]: 121(2620) CRITICAL: <core> [core/pass_fd.c:277]: receive_fd(): EOF on 26
Nov 29 15:47:05 modcom-sbc-dev1 kamailio[2497]: 121(2620) DEBUG: <core> [core/tcp_main.c:3512]: handle_ser_child(): dead child 3, pid 2502 (shutting down?)
Nov 29 15:47:05 modcom-sbc-dev1 kamailio[2497]: 121(2620) DEBUG: <core> [core/io_wait.h:602]: io_watch_del(): DBG: io_watch_del (0x55ae7132ffc0, 26, -1, 0x0) fd_no=126 called
Nov 29 15:47:05 modcom-sbc-dev1 kamailio[2497]:  0(2499) ALERT: <core> [main.c:741]: handle_sigs(): child process 2502 exited normally, status=255
Nov 29 15:47:05 modcom-sbc-dev1 kamailio[2497]:  0(2499) INFO: <core> [main.c:771]: handle_sigs(): terminating due to SIGCHLD
Nov 29 15:47:05 modcom-sbc-dev1 kamailio[2497]:  0(2499) DEBUG: <core> [main.c:773]: handle_sigs(): terminating due to SIGCHLD
Nov 29 15:47:05 modcom-sbc-dev1 kamailio[2497]: 118(2617) INFO: <core> [main.c:826]: sig_usr(): signal 15 received

Let me know if I can somehow help dissect this issue, I don't really know where to start. Thanks.

BR,
George





_______________________________________________
Kamailio (SER) - Development Mailing List
[hidden email]
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

-- 
Daniel-Constantin Mierla
www.twitter.com/miconda -- www.linkedin.com/in/miconda
Kamailio Advanced Training - www.asipto.com
Kamailio World Conference - May 14-16, 2018 - www.kamailioworld.com

_______________________________________________
Kamailio (SER) - Development Mailing List
[hidden email]
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
Reply | Threaded
Open this post in threaded view
|

Re: Kamailio 5.1 nightly (rc2) crashing - udp_rcv_loop(): print buffer building failed

Daniel-Constantin Mierla

I pushed a patch few days ago related to this report, if you still get the issue, better open an issue on bug tracker -- it is safer not to forget about it.

Cheers,
Daniel


On 30.11.17 20:04, Daniel-Constantin Mierla wrote:

Hello,

did you run with debug=3 in kamailio.cfg? That piece of code is new in 5.1, but executed only when debug is at least 3 (which is for troubleshooting purposes, not for production). Anyhow, I will check a bit that part to see if anything is wrong there...

Cheers,
Daniel


On 30.11.17 18:52, George Diamantopoulos wrote:
OK, so I performed a clean installation on a new debian stretch instance I haven't been able to reproduce the issue so far. Either something changed in the latest nightly, or the fact that I upgraded from 5.0.4 was causing the issue.

The deployment is automated with ansible, so there shouldn't be any deviations in the configuration of the host. Only other difference (apart from the clean kamailio install) should be rtpengine's building, which was performed anew. However, I don't see how that might have affected things (after all, I don't believe kamailio is a build time dependency for rtpengine)...

On 30 November 2017 at 01:08, George Diamantopoulos <[hidden email]> wrote:
Oh, I also tried with 5.2.0-dev1 nightly and it also crashes in the same way...

On 30 November 2017 at 01:07, George Diamantopoulos <[hidden email]> wrote:
I've been trying to get a core dump on this for hours now, but I just can't get one! Is there anything special needed with this build on debian stretch? Checked ulimits, disable_core_dump, /etc/default/kamailio, started it as root (-u root -g root), from systemd with user kamailio, with working directory set in command line and in config file, I just can't get a core dump...

Thanks

On 29 November 2017 at 18:10, George Diamantopoulos <[hidden email]> wrote:
Hello,

I've just installed kamailio 5.1 nightly from the debian repo for stretch. I didn't change anything in the configuration compared to 5.0.4 (which I was testing previously), apart from the database structure as found in the wiki for upgrading from 5.0.x.

Now kamailio crashes after a while, with the following output in the logs:

Nov 29 15:47:05 modcom-sbc-dev1 kamailio[2497]:  3(2502) ERROR: <core> [core/udp_server.c:481]: udp_rcv_loop(): print buffer building failed
Nov 29 15:47:05 modcom-sbc-dev1 kamailio[2497]: 121(2620) CRITICAL: <core> [core/pass_fd.c:277]: receive_fd(): EOF on 26
Nov 29 15:47:05 modcom-sbc-dev1 kamailio[2497]: 121(2620) DEBUG: <core> [core/tcp_main.c:3512]: handle_ser_child(): dead child 3, pid 2502 (shutting down?)
Nov 29 15:47:05 modcom-sbc-dev1 kamailio[2497]: 121(2620) DEBUG: <core> [core/io_wait.h:602]: io_watch_del(): DBG: io_watch_del (0x55ae7132ffc0, 26, -1, 0x0) fd_no=126 called
Nov 29 15:47:05 modcom-sbc-dev1 kamailio[2497]:  0(2499) ALERT: <core> [main.c:741]: handle_sigs(): child process 2502 exited normally, status=255
Nov 29 15:47:05 modcom-sbc-dev1 kamailio[2497]:  0(2499) INFO: <core> [main.c:771]: handle_sigs(): terminating due to SIGCHLD
Nov 29 15:47:05 modcom-sbc-dev1 kamailio[2497]:  0(2499) DEBUG: <core> [main.c:773]: handle_sigs(): terminating due to SIGCHLD
Nov 29 15:47:05 modcom-sbc-dev1 kamailio[2497]: 118(2617) INFO: <core> [main.c:826]: sig_usr(): signal 15 received

Let me know if I can somehow help dissect this issue, I don't really know where to start. Thanks.

BR,
George





_______________________________________________
Kamailio (SER) - Development Mailing List
[hidden email]
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

-- 
Daniel-Constantin Mierla
www.twitter.com/miconda -- www.linkedin.com/in/miconda
Kamailio Advanced Training - www.asipto.com
Kamailio World Conference - May 14-16, 2018 - www.kamailioworld.com

-- 
Daniel-Constantin Mierla
www.twitter.com/miconda -- www.linkedin.com/in/miconda
Kamailio Advanced Training - www.asipto.com
Kamailio World Conference - May 14-16, 2018 - www.kamailioworld.com

_______________________________________________
Kamailio (SER) - Development Mailing List
[hidden email]
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
Reply | Threaded
Open this post in threaded view
|

Re: Kamailio 5.1 nightly (rc2) crashing - udp_rcv_loop(): print buffer building failed

George Diamantopoulos
I know it sounds scary, but unfortunately I haven't been able to reproduce the issue since I did a clean installation. I'll definitely post again if it somehow manifests...

BR,
George

On 4 December 2017 at 11:00, Daniel-Constantin Mierla <[hidden email]> wrote:

I pushed a patch few days ago related to this report, if you still get the issue, better open an issue on bug tracker -- it is safer not to forget about it.

Cheers,
Daniel


On 30.11.17 20:04, Daniel-Constantin Mierla wrote:

Hello,

did you run with debug=3 in kamailio.cfg? That piece of code is new in 5.1, but executed only when debug is at least 3 (which is for troubleshooting purposes, not for production). Anyhow, I will check a bit that part to see if anything is wrong there...

Cheers,
Daniel


On 30.11.17 18:52, George Diamantopoulos wrote:
OK, so I performed a clean installation on a new debian stretch instance I haven't been able to reproduce the issue so far. Either something changed in the latest nightly, or the fact that I upgraded from 5.0.4 was causing the issue.

The deployment is automated with ansible, so there shouldn't be any deviations in the configuration of the host. Only other difference (apart from the clean kamailio install) should be rtpengine's building, which was performed anew. However, I don't see how that might have affected things (after all, I don't believe kamailio is a build time dependency for rtpengine)...

On 30 November 2017 at 01:08, George Diamantopoulos <[hidden email]> wrote:
Oh, I also tried with 5.2.0-dev1 nightly and it also crashes in the same way...

On 30 November 2017 at 01:07, George Diamantopoulos <[hidden email]> wrote:
I've been trying to get a core dump on this for hours now, but I just can't get one! Is there anything special needed with this build on debian stretch? Checked ulimits, disable_core_dump, /etc/default/kamailio, started it as root (-u root -g root), from systemd with user kamailio, with working directory set in command line and in config file, I just can't get a core dump...

Thanks

On 29 November 2017 at 18:10, George Diamantopoulos <[hidden email]> wrote:
Hello,

I've just installed kamailio 5.1 nightly from the debian repo for stretch. I didn't change anything in the configuration compared to 5.0.4 (which I was testing previously), apart from the database structure as found in the wiki for upgrading from 5.0.x.

Now kamailio crashes after a while, with the following output in the logs:

Nov 29 15:47:05 modcom-sbc-dev1 kamailio[2497]:  3(2502) ERROR: <core> [core/udp_server.c:481]: udp_rcv_loop(): print buffer building failed
Nov 29 15:47:05 modcom-sbc-dev1 kamailio[2497]: 121(2620) CRITICAL: <core> [core/pass_fd.c:277]: receive_fd(): EOF on 26
Nov 29 15:47:05 modcom-sbc-dev1 kamailio[2497]: 121(2620) DEBUG: <core> [core/tcp_main.c:3512]: handle_ser_child(): dead child 3, pid 2502 (shutting down?)
Nov 29 15:47:05 modcom-sbc-dev1 kamailio[2497]: 121(2620) DEBUG: <core> [core/io_wait.h:602]: io_watch_del(): DBG: io_watch_del (0x55ae7132ffc0, 26, -1, 0x0) fd_no=126 called
Nov 29 15:47:05 modcom-sbc-dev1 kamailio[2497]:  0(2499) ALERT: <core> [main.c:741]: handle_sigs(): child process 2502 exited normally, status=255
Nov 29 15:47:05 modcom-sbc-dev1 kamailio[2497]:  0(2499) INFO: <core> [main.c:771]: handle_sigs(): terminating due to SIGCHLD
Nov 29 15:47:05 modcom-sbc-dev1 kamailio[2497]:  0(2499) DEBUG: <core> [main.c:773]: handle_sigs(): terminating due to SIGCHLD
Nov 29 15:47:05 modcom-sbc-dev1 kamailio[2497]: 118(2617) INFO: <core> [main.c:826]: sig_usr(): signal 15 received

Let me know if I can somehow help dissect this issue, I don't really know where to start. Thanks.

BR,
George





_______________________________________________
Kamailio (SER) - Development Mailing List
[hidden email]
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

-- 
Daniel-Constantin Mierla
www.twitter.com/miconda -- www.linkedin.com/in/miconda
Kamailio Advanced Training - www.asipto.com
Kamailio World Conference - May 14-16, 2018 - www.kamailioworld.com

-- 
Daniel-Constantin Mierla
www.twitter.com/miconda -- www.linkedin.com/in/miconda
Kamailio Advanced Training - www.asipto.com
Kamailio World Conference - May 14-16, 2018 - www.kamailioworld.com


_______________________________________________
Kamailio (SER) - Development Mailing List
[hidden email]
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev