bummer [Elli Mylonas Senior Digital Humanities Librarian and Center for Digital Scholarship University Library Brown University library.brown.edu/cds] On Thu, Apr 27, 2017 at 11:37 AM, Peter Stadler wrote:

I can confirm that it seems to be down from my end as well.

Best Peter

...

Am 27.04.2017 um 10:32 schrieb Mylonas, Elli :

Hi all - we are at the Dixit workshop in Athens, and it appears that tei-c is inaccessible from our classroom, also from my Brown VPN. Roma, too.

Is it just us? did I miss a scheduled downtime message?

just checking, and thank you. --elli -- tei-council mailing list tei-council@lists.tei-c.org http://lists.lists.tei-c.org/mailman/listinfo/tei-council

PLEASE NOTE: postings to this list are publicly archived

-- tei-council mailing list tei-council@lists.tei-c.org http://lists.lists.tei-c.org/mailman/listinfo/tei-council

PLEASE NOTE: postings to this list are publicly archived

Hello All, Thank you for the messages and your patience as we work through this. I made some adjustments this morning. How is performance now? ------------------------------------- TECHNICAL DETAILS ------------------------------------- *Overview* The server was using all available memory (RAM) and starting to "swap" (use disk space for RAM, which is slow). The server did not run out of swap so it did not fully crash. CPU usage also spiked, but I believe that is simply because of the RAM utilization. The question is why was it using so much RAM. Nothing seems to be excessively heavy usage so it might just be cumulative adding to what's running on the server. It's probably been a while since I looked closely at memory utilization and I believe it has been increasing over time. *Apache HTTPD* Apache did reach the max child processes I had previously set. I previously calculated the number of child processes based on how much memory they were using and how much memory was available on the server. Interestingly the HTTPD processes seem to be slightly bigger than they used to be -- this alone was using nearly all available RAM. Nothing stood out as particularly noticeable in the Apache logs. The action I've taken is to adjust the Apache HTTPD conf by decreasing the timeouts and the max number of child processes. Decreasing the max number of child processes will hopefully make us not run out of RAM and start swapping. Decreasing the timeouts should mean processes don't hang around as long so it should decrease the need for as many child processes. I made that change earlier this morning and currently I'm only seeing significantly less HTTPD children so this might have made a notable impact. *Tomcat* In addition to everything running out of the standard Apache HTTPD instance (static sites, WordPress, Drupal, etc.) there are 7 standalone java (tomcat) applications that run on the server. They currently each have their own Tomcat instance which uses anywhere from 500mb to 2gb of RAM each. That means at a min. they are using 3.5GB of RAM and sometimes more. We could dig into each Tomcat config to limit how much max RAM it can use, though I don't necessarily know how much each one needs to function. We could combine into one Tomcat instance but I don't know how nicely they will play with each other. As one example, the TEI CMS, currently is configured in Tomcat to not use more than 1.5GB of memory. This is reasonable for Tomcat but even that is 10% of the total server RAM (and TEI uses RAM for other processes like Apache HTTPD as well). *Next Steps* Let's see how things function with the tweaks to Apache HTTPD I have made. In addition, perhaps we can try to limit how much memory each Tomcat instance uses and going forward try to limit the amount of standalone Tomcat applications. Thanks, Ian On Thu, Apr 27, 2017 at 5:11 AM, James Cummings wrote:

Seems up but really slow here.

The current load averages on the machine were load average: 10.99, 9.19, 9.20 with httpd java and mysqld taking up a fair amount of CPU usage when I ran 'top'. I don't think I have permissions to restart anything.

According to ~/logs/error_log there seems to be a lot of hammering of Roma happening.

But also just timeouts on the main website.

The sites which we share hosting with, adho, digitalhumanities, etc are also having the same problem.

-James

On 27/04/17 09:41, Mylonas, Elli wrote:

...

bummer

[Elli Mylonas Senior Digital Humanities Librarian and Center for Digital Scholarship University Library Brown University library.brown.edu/cds]

On Thu, Apr 27, 2017 at 11:37 AM, Peter Stadler wrote:

I can confirm that it seems to be down from my end as well.

...

Best Peter

Am 27.04.2017 um 10:32 schrieb Mylonas, Elli :

...

Hi all - we are at the Dixit workshop in Athens, and it appears that

tei-c

...

is inaccessible from our classroom, also from my Brown VPN. Roma, too.

Is it just us? did I miss a scheduled downtime message?

just checking, and thank you. --elli -- tei-council mailing list tei-council@lists.tei-c.org http://lists.lists.tei-c.org/mailman/listinfo/tei-council

PLEASE NOTE: postings to this list are publicly archived

-- tei-council mailing list tei-council@lists.tei-c.org http://lists.lists.tei-c.org/mailman/listinfo/tei-council

PLEASE NOTE: postings to this list are publicly archived

-- Dr James Cummings, James.Cummings@it.ox.ac.uk Academic IT Services, University of Oxford

-- Ian Rifkin '04, MS '09 Data Warehouse Architect Information Technology Services (ITS) Brandeis University http://go.brandeis.edu/u:irifkin