[sr-dev] Multiple memleaks and segfauts in Kamailio v5.0.4

classic Classic list List threaded Threaded
7 messages Options
M S
Reply | Threaded
Open this post in threaded view
|

[sr-dev] Multiple memleaks and segfauts in Kamailio v5.0.4

M S
Hi,

I have kamailio v5.0.4 git rev. 5f26a4e8c55cdf5a5a7eaf6864c4f5213d321293 installed on 32bit Debian Wheezy virtual machine (1GB RAM and 1 vCPU) for development and testing of a project.

We need to inspect sip traces to / from webrtc endpoint on kamailio side, for which we use siptrace module with following config,


# ----- siptrace params -----
modparam("siptrace", "trace_on", 1)
modparam("siptrace", "hep_mode_on", 0)
modparam("siptrace", "xheaders_write", 1)
modparam("siptrace", "trace_to_database", 0)
modparam("siptrace", "trace_flag", FLT_SIP_TRACE)
modparam("siptrace", "traced_user_avp", "$avp(traced_user)")
modparam("siptrace", "duplicate_uri", "WEBRTC_SIP_TRACE_ADDRESS")


We observed multiple problems of memleaks and segfaults, here is a summary,

1. Calling sip_trace method without any arguements or with only one arguement (duplicate uri) crashes kamailio with segfault.

https://kamailio.org/docs/modules/5.0.x/modules/siptrace.html#siptrace.f.sip_trace

2. To avoid segfault, either I have to specify both args (i am not sure what to set as 2nd arg, so i used "$ci" for it) OR disable "xheaders_write" module param. However, this causes slow memory leak and in about 48 hours kamailio runs out of memory (sip traffic is less then 10 concurrent users)

https://kamailio.org/docs/modules/5.0.x/modules/siptrace.html#siptrace.p.xheaders_write

3. If we remove siptrace module and use "network:msg" event route in corex module with "send_data" method then we ran in to PV buffer overflow problem (webrtc invites are very big > 10KB). So, I increased "pv_buffer_size" to 16K and "pv_buffer_slots" to 12. Which again crashes kamailio with segfault.

https://kamailio.org/docs/modules/5.0.x/modules/corex.html#async.evr.network_io

https://kamailio.org/docs/modules/5.0.x/modules/corex.html#corex.f.send_data

http://www.kamailio.org/wiki/cookbooks/4.1.x/core#pv_buffer_size

http://www.kamailio.org/wiki/cookbooks/4.1.x/core#pv_buffer_slots


Removing "pv_buffer_slots" from config fixes the crash problem, and right now we are observing kamailio if stays stable.


Thank you.



_______________________________________________
Kamailio (SER) - Development Mailing List
[hidden email]
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
Reply | Threaded
Open this post in threaded view
|

Re: Multiple memleaks and segfauts in Kamailio v5.0.4

Daniel-Constantin Mierla

Hello,

if you get a crash, then take the backtrace from the corefile with gdb (inside the gdb session inspecting the corefile, do "bt full"), it is very useful information to discover where the crash happens.

It is also better to open on issue on bug tracker, so the investigation of the crash can be done easier as well as reduce the risck of forgetting about the report:

  - https://github.com/kamailio/kamailio/issues

Cheers,
Daniel

On 03.12.17 18:41, M S wrote:
Hi,

I have kamailio v5.0.4 git rev. 5f26a4e8c55cdf5a5a7eaf6864c4f5213d321293 installed on 32bit Debian Wheezy virtual machine (1GB RAM and 1 vCPU) for development and testing of a project.

We need to inspect sip traces to / from webrtc endpoint on kamailio side, for which we use siptrace module with following config,


# ----- siptrace params -----
modparam("siptrace", "trace_on", 1)
modparam("siptrace", "hep_mode_on", 0)
modparam("siptrace", "xheaders_write", 1)
modparam("siptrace", "trace_to_database", 0)
modparam("siptrace", "trace_flag", FLT_SIP_TRACE)
modparam("siptrace", "traced_user_avp", "$avp(traced_user)")
modparam("siptrace", "duplicate_uri", "WEBRTC_SIP_TRACE_ADDRESS")


We observed multiple problems of memleaks and segfaults, here is a summary,

1. Calling sip_trace method without any arguements or with only one arguement (duplicate uri) crashes kamailio with segfault.

https://kamailio.org/docs/modules/5.0.x/modules/siptrace.html#siptrace.f.sip_trace

2. To avoid segfault, either I have to specify both args (i am not sure what to set as 2nd arg, so i used "$ci" for it) OR disable "xheaders_write" module param. However, this causes slow memory leak and in about 48 hours kamailio runs out of memory (sip traffic is less then 10 concurrent users)

https://kamailio.org/docs/modules/5.0.x/modules/siptrace.html#siptrace.p.xheaders_write

3. If we remove siptrace module and use "network:msg" event route in corex module with "send_data" method then we ran in to PV buffer overflow problem (webrtc invites are very big > 10KB). So, I increased "pv_buffer_size" to 16K and "pv_buffer_slots" to 12. Which again crashes kamailio with segfault.

https://kamailio.org/docs/modules/5.0.x/modules/corex.html#async.evr.network_io

https://kamailio.org/docs/modules/5.0.x/modules/corex.html#corex.f.send_data

http://www.kamailio.org/wiki/cookbooks/4.1.x/core#pv_buffer_size

http://www.kamailio.org/wiki/cookbooks/4.1.x/core#pv_buffer_slots


Removing "pv_buffer_slots" from config fixes the crash problem, and right now we are observing kamailio if stays stable.


Thank you.




_______________________________________________
Kamailio (SER) - Development Mailing List
[hidden email]
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

-- 
Daniel-Constantin Mierla
www.twitter.com/miconda -- www.linkedin.com/in/miconda
Kamailio Advanced Training - www.asipto.com
Kamailio World Conference - May 14-16, 2018 - www.kamailioworld.com

_______________________________________________
Kamailio (SER) - Development Mailing List
[hidden email]
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
Reply | Threaded
Open this post in threaded view
|

[sr-dev] Multiple memleaks and segfauts in Kamailio v5.0.4

Juha Heinanen
In reply to this post by M S
M S writes:

> 1. Calling sip_trace method without any arguements or with only one
> arguement (duplicate uri) crashes kamailio with segfault.

My K does not crash when I make call like this:

    setflag(TRACE);
    sip_trace();

but I have not defined as many vars as you, but only these:

# -- siptrace params
modparam("siptrace", "db_url", "xxx")
modparam("siptrace", "trace_flag", TRACE)

-- Juha

_______________________________________________
Kamailio (SER) - Development Mailing List
[hidden email]
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
M S
Reply | Threaded
Open this post in threaded view
|

Re: Multiple memleaks and segfauts in Kamailio v5.0.4

M S
Thanks @Daniel, i am opening up the tickets.

@Juha, we are not dumping sip trace to db but rather to a udp socket (typically 127.0.0.1:5060), where we can observer them using ngrep or sipgrep cli utilities. The whole purpose is to watch sip trace at runtime while conducting our tests and observe the changes as sip client changes its behaviour (it is sort of whitebox testing). Dumping in db or Homer is bit of waste of time and resources for our work. That's why we disabled both of these via module params. We have been doing it with kamailio v4.x for last 1-2 years or so and it never caused any trouble. These problems started happening only after upgrading our dev setup to v5.0.

Thank you.



On Mon, Dec 4, 2017 at 2:54 AM, Juha Heinanen <[hidden email]> wrote:
M S writes:

> 1. Calling sip_trace method without any arguements or with only one
> arguement (duplicate uri) crashes kamailio with segfault.

My K does not crash when I make call like this:

    setflag(TRACE);
    sip_trace();

but I have not defined as many vars as you, but only these:

# -- siptrace params
modparam("siptrace", "db_url", "xxx")
modparam("siptrace", "trace_flag", TRACE)

-- Juha

_______________________________________________
Kamailio (SER) - Development Mailing List
[hidden email]
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev


_______________________________________________
Kamailio (SER) - Development Mailing List
[hidden email]
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
M S
Reply | Threaded
Open this post in threaded view
|

Re: Multiple memleaks and segfauts in Kamailio v5.0.4

M S
For "pv_buffer_slots" issue, kamailio crashes during startup and there is no core dump. What should i do about it?

Here is what i see in logs (just one log line).

--
Dec  4 03:44:19 webrtc-as-rct kernel: [1814300.507165] kamailio[16879] segfault at c0c0c0c7 ip b7700239 sp bfd9a0f8 error 7 in libc-2.13.so[b75e2000+161000]
--


Thank you.



On Mon, Dec 4, 2017 at 3:14 AM, M S <[hidden email]> wrote:
Thanks @Daniel, i am opening up the tickets.

@Juha, we are not dumping sip trace to db but rather to a udp socket (typically 127.0.0.1:5060), where we can observer them using ngrep or sipgrep cli utilities. The whole purpose is to watch sip trace at runtime while conducting our tests and observe the changes as sip client changes its behaviour (it is sort of whitebox testing). Dumping in db or Homer is bit of waste of time and resources for our work. That's why we disabled both of these via module params. We have been doing it with kamailio v4.x for last 1-2 years or so and it never caused any trouble. These problems started happening only after upgrading our dev setup to v5.0.

Thank you.



On Mon, Dec 4, 2017 at 2:54 AM, Juha Heinanen <[hidden email]> wrote:
M S writes:

> 1. Calling sip_trace method without any arguements or with only one
> arguement (duplicate uri) crashes kamailio with segfault.

My K does not crash when I make call like this:

    setflag(TRACE);
    sip_trace();

but I have not defined as many vars as you, but only these:

# -- siptrace params
modparam("siptrace", "db_url", "xxx")
modparam("siptrace", "trace_flag", TRACE)

-- Juha

_______________________________________________
Kamailio (SER) - Development Mailing List
[hidden email]
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev



_______________________________________________
Kamailio (SER) - Development Mailing List
[hidden email]
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
Reply | Threaded
Open this post in threaded view
|

Re: Multiple memleaks and segfauts in Kamailio v5.0.4

Daniel-Constantin Mierla

I think I catch this one, too -- I pushed a patch for it to branch 5.0 as well.

Cheers,
Daniel


On 04.12.17 03:49, M S wrote:
For "pv_buffer_slots" issue, kamailio crashes during startup and there is no core dump. What should i do about it?

Here is what i see in logs (just one log line).

--
Dec  4 03:44:19 webrtc-as-rct kernel: [1814300.507165] kamailio[16879] segfault at c0c0c0c7 ip b7700239 sp bfd9a0f8 error 7 in libc-2.13.so[b75e2000+161000]
--


Thank you.



On Mon, Dec 4, 2017 at 3:14 AM, M S <[hidden email]> wrote:
Thanks @Daniel, i am opening up the tickets.

@Juha, we are not dumping sip trace to db but rather to a udp socket (typically 127.0.0.1:5060), where we can observer them using ngrep or sipgrep cli utilities. The whole purpose is to watch sip trace at runtime while conducting our tests and observe the changes as sip client changes its behaviour (it is sort of whitebox testing). Dumping in db or Homer is bit of waste of time and resources for our work. That's why we disabled both of these via module params. We have been doing it with kamailio v4.x for last 1-2 years or so and it never caused any trouble. These problems started happening only after upgrading our dev setup to v5.0.

Thank you.



On Mon, Dec 4, 2017 at 2:54 AM, Juha Heinanen <[hidden email]> wrote:
M S writes:

> 1. Calling sip_trace method without any arguements or with only one
> arguement (duplicate uri) crashes kamailio with segfault.

My K does not crash when I make call like this:

    setflag(TRACE);
    sip_trace();

but I have not defined as many vars as you, but only these:

# -- siptrace params
modparam("siptrace", "db_url", "xxx")
modparam("siptrace", "trace_flag", TRACE)

-- Juha

_______________________________________________
Kamailio (SER) - Development Mailing List
[hidden email]
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev




_______________________________________________
Kamailio (SER) - Development Mailing List
[hidden email]
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

-- 
Daniel-Constantin Mierla
www.twitter.com/miconda -- www.linkedin.com/in/miconda
Kamailio Advanced Training - www.asipto.com
Kamailio World Conference - May 14-16, 2018 - www.kamailioworld.com

_______________________________________________
Kamailio (SER) - Development Mailing List
[hidden email]
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
M S
Reply | Threaded
Open this post in threaded view
|

Re: Multiple memleaks and segfauts in Kamailio v5.0.4

M S
Super. I will test it and report back.

Many thanks and kind regards.


On Mon, Dec 4, 2017 at 9:48 AM, Daniel-Constantin Mierla <[hidden email]> wrote:

I think I catch this one, too -- I pushed a patch for it to branch 5.0 as well.

Cheers,
Daniel


On 04.12.17 03:49, M S wrote:
For "pv_buffer_slots" issue, kamailio crashes during startup and there is no core dump. What should i do about it?

Here is what i see in logs (just one log line).

--
Dec  4 03:44:19 webrtc-as-rct kernel: [1814300.507165] kamailio[16879] segfault at c0c0c0c7 ip b7700239 sp bfd9a0f8 error 7 in libc-2.13.so[b75e2000+161000]
--


Thank you.



On Mon, Dec 4, 2017 at 3:14 AM, M S <[hidden email]> wrote:
Thanks @Daniel, i am opening up the tickets.

@Juha, we are not dumping sip trace to db but rather to a udp socket (typically 127.0.0.1:5060), where we can observer them using ngrep or sipgrep cli utilities. The whole purpose is to watch sip trace at runtime while conducting our tests and observe the changes as sip client changes its behaviour (it is sort of whitebox testing). Dumping in db or Homer is bit of waste of time and resources for our work. That's why we disabled both of these via module params. We have been doing it with kamailio v4.x for last 1-2 years or so and it never caused any trouble. These problems started happening only after upgrading our dev setup to v5.0.

Thank you.



On Mon, Dec 4, 2017 at 2:54 AM, Juha Heinanen <[hidden email]> wrote:
M S writes:

> 1. Calling sip_trace method without any arguements or with only one
> arguement (duplicate uri) crashes kamailio with segfault.

My K does not crash when I make call like this:

    setflag(TRACE);
    sip_trace();

but I have not defined as many vars as you, but only these:

# -- siptrace params
modparam("siptrace", "db_url", "xxx")
modparam("siptrace", "trace_flag", TRACE)

-- Juha

_______________________________________________
Kamailio (SER) - Development Mailing List
[hidden email]
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev




_______________________________________________
Kamailio (SER) - Development Mailing List
[hidden email]
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

-- 
Daniel-Constantin Mierla
www.twitter.com/miconda -- www.linkedin.com/in/miconda
Kamailio Advanced Training - www.asipto.com
Kamailio World Conference - May 14-16, 2018 - www.kamailioworld.com


_______________________________________________
Kamailio (SER) - Development Mailing List
[hidden email]
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev