On Sun, Feb 9, 2014 at 6:04 AM, Robin Bowes <robin.bowes_at_yo61.com> wrote:
> On Sat, 2014-02-08 at 22:32 -0800, Kevin Burke wrote:
> > Hello,
> > Recently I set up a daemon to monitor server load, that's monitored by
> > runit. I ran into a few problems that ended up taking the better part of
> a
> > day to debug, so I wanted to note them here.
> >
> > *Despite starting a single Python process via runit, after about 20
> minutes
> > several Python processes would be running on the host.* The problem, it
> > turns out, was that I wasn't using exec() to replace the `run` script
> with
> > my Python process. It may be worth noting this in the FAQ; something like
> > "I started my service with runit, but it looks like several instances are
> > running" and tell people to ensure they started their service with exec.
> It
> > may also be worth telling people that if monitored via top() or similar,
> > your running program must appear as a child of runsvdir for runsv to
> > effectively monitor it.
>
> This is already covered in the FAQ:
>
> http://smarden.org/runit/faq.html#create
>
>
> > *Second, changes to my `run` script weren't being picked up by runs v. *I
> > havne't double checked this, but I believe running "sv down <service>; sv
> > up <service>" was not enough for runit to pick up changes in my program's
> > `run` file. Only after I removed the /service/myservice directory were
> the
> > changes picked up. Again, I haven't double checked this but if this is
> > actually the program's behavior then it may be worth noting this in a
> FAQ.
>
> I am not 100% sure of this but I think the correct way to pickup changes
> in the run script is to exit the service and start it again, ie. do
> this:
>
> sv exit your_service
> sv start your_service
runit doesn't actually cache your run script. The "run" script can actually
be any executable, and runit exec's it. There should never be any problem
loading old versions, since it's actually executed by the operating system.
I'm often adding new services and tweaking them and I've never had an
issue running older versions, even with
sv restart your_service
At least on linux, it all works as expected.
> > *Third, does runsv log anywhere*, or can it be run in a debug mode? When
> I
> > saw multiple processes popping up in the process list, my intuition was
> to
> > try and enable debug logs to figure out what was doing on.
>
> For runsvdir logging:
>
> http://smarden.org/runit/runsvdir.8.html
>
> To create a service with an appendant log service:
>
> http://smarden.org/runit/faq.html#createlog
>
> > Finally, I would like to restart the process if it is using too much
> > memory. Reading the `chpst` man page reveals this:
> >
> > *-m bytes:* limit memory. Limit the data segment, stack segment, locked
> > physical pages, and total of all segment per process to bytes bytes each.
> >
> > What is the actual behavior if the process tries to use more memory than
> it
> > is allowed? Will it (can it) just restart the process?
>
> I believe what happens is that chpst enforces the memory limit and the
> limited program will simply die if it tries to allocate more memory than
> it is allowed. runsv will then restart the program.
I've taken advantage of this exact feature before and it's wonderful. The
program actually receives an error code telling it that it cannot have the
memory that it's asking for when it runs out. In C using malloc this means
a NULL pointer result and errno set appropriately. In C++ using new you
will get a std::bad_alloc exception thrown.
Many programs don't handle this all that well, and just assume that they
will always get all the memory they've requested, in which case the above
true, but only because of the program author trying to use a null reference
or not handling an exception.
In practice, this means that when writing a program you can plan for this
feature and use that as an opportunity to free up some memory and keep your
program within the specified bounds.
>
> R.
>
> > Thanks for your help! runsv/chpst are awesome tools and I'm glad they are
> > so good.
> >
> > Kevin
> >
> > --
> > Kevin Burke | Twilio
> > phone: 925.271.7005 | kev.inburke.com
>
>
>
>
Received on Sun Feb 09 2014 - 15:44:27 UTC