![]() If you need more help, you can submit a support request from the Azure portal.Connecting the Azure community to the right resources: answers, support, and experts. Connect with - the official Microsoft Azure account for improving customer experience.Get answers from Azure experts through Azure Community Support.If you didn't see your problem or are unable to solve your issue, visit one of the following channels for more support: Look for the other symptoms outlined in this document.This exception also indicates that the zookeeper client is ending sessions prematurely.This exception usually means that the client is no longer active and the server is unable to send a message.This exception will be seen on the zookeeper servers (/var/log/zookeeper/zookeeper-zookeeper-* or /var/log/hdinsight-zookeeper/zookeeper* files).Do not purge snapshots manually - deleting snapshots manually could result in data lossĬancelledKeyException in the zookeeper server log doesn't require snapshot cleanup.sudo python /opt/startup_scripts/startup_hdinsight_zookeeper.py.sudo lsof -i :2182 will give you the process ID to kill.Stop and restart HDInsight zookeeper manually.Hadoop zookeeper config can be updated and the service can be restarted through Ambari.Set autopurge.snapRetainCount to a value of 3 and restart the zookeeper servers./etc/hdinsight-zookeeper/conf/zoo.cfg for HDInsight zookeeper./etc/zookeeper/conf/zoo.cfg for Hadoop zookeeper.This property can be found in the following files: The number of snapshots that are retained, is controlled by the configuration key autopurge.snapRetainCount.By default, the last 30 snapshots are retained.Zookeepers are configured to auto purge old snapshots.If there are any hourly jobs running at this time, randomize the start time across different zookeeper servers.Log in to the zookeeper server and check the /etc/crontab.If the servers are running, the result will include statics of client connections and other statistics.If the command shows no output, then it means that the zookeeper servers are not running.Port 2182 is used by the HDInsight zookeeper (to provide HA for services that are not natively HA).Port 2181 is the apache zookeeper instance.Find the zookeeper servers from the /etc/hosts file or from Ambari UI.This could result in quorum loss, frequent failovers, and other issues.In the logs for Resource Manager, Namenode and others, you will see frequent client connection timeouts.Zookeeper clients are reporting frequent timeouts. ![]() In the Ambari UI, if you see near 100% sustained CPU usage on the zookeeper servers, then the zookeeper sessions open during that time can expire and time out.High CPU usage on the zookeeper servers.Jobs can fail temporarily due to Zookeeper connection issues.Make sure that the issue happens repeatedly (do not use these solutions for one off cases).Confirm from the logs that it is related to Zookeeper connections. ![]()
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |