![]() Like SQL You can Kill this job from Resource Manager it self – Click on Job and on top you will find a button for killing same query, like this job is running for last 7 hours, you can Kill it by clicking Kill Applicationįeel free to add comments, may be in next post I can take a deep dive into each log and action.I can further look into job by clicking into job id (Job ID > Logs and Pull query from there dag) and check how’s query look like and what’s its doing under the hood – is it scanning TBs of partitions or filtering on non-partitioned fields or what. In this case filtered job may be problem making query which is causing other jobs to wait in queue as its consuming 60% of resource and 2.2 TB of total 3.75TB allocated for this cluster.In following screenshot – you can notice multiple options for checking Jobs like in this case I have this specific job which is consuming:.Or if you don’t want to go through the whole hassle just type in popup provide security information admin and its password – and it would open Resource Manager – it provides you all information about jobs (running/pending/finished/etc.) on HDI cluster ![]() To open the UI, click Quick links and then click Spark History Server. Click Yarn > Quick Links > Active Head Node >Resource Manager UI Figure 4-30 Ambari Dashboard: Apache Spark service - Summary tab - Apache Spark.Successful login will open Ambari dashboard which looks something like this:.Open following link of HDInsight Cluster ( ).Objective: How to find long running or resource intensive Hive query? Coming from SQL background where you have dependency on SQL Profiler Trace for understanding what’s going on SQL Server like who is running non optimized long query, where its scanning through all partitions and yes who is using non indexed columns for filter right? So coming to HDInsight world wondering do we have something like SQL Profiler for tracking bad jobs – Yes we do, as a part of Apache Ambari Project – (as per wiki) Whole objective of Ambari is to make Hadoop management easy by providing a mechanism of managing or monitoring Hadoop clusters with it’s easy to use UI, in this Post I will show how to use Ambari for getting long running or resource intensive jobs.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |