{"id":192956,"date":"2021-08-08T12:00:33","date_gmt":"2021-08-08T19:00:33","guid":{"rendered":"https:\/\/m2msupport.net\/m2msupport\/?page_id=192956"},"modified":"2021-08-08T12:00:37","modified_gmt":"2021-08-08T19:00:37","slug":"get-a-list-of-dataflow-jobs-in-google-cloud-platform-gcp","status":"publish","type":"page","link":"https:\/\/m2msupport.net\/m2msupport\/get-a-list-of-dataflow-jobs-in-google-cloud-platform-gcp\/","title":{"rendered":"Get a list of dataflow jobs in Google Cloud Platform (GCP)"},"content":{"rendered":"<p>Google Cloud Dataflow is data processing service that can be used for streaming and batch applications. Users can setup pipelines in Dataflow to integrate and process large datasets.<\/p>\n<p>With pub\/sub, users can setup dataflow pipelines to write messages from a pub\/sub topic or subscription to a BigQuery table.<\/p>\n<p><a href=\"https:\/\/m2msupport.net\/m2msupport\/download-iot-cloud-tester\/\">IoT Cloud Tester<\/a>&nbsp; application provides an easy interface to get a list of dataflow jobs in Google Cloud Platform.<\/p>\n<h1>To get a list of dataflow jobs in Google Cloud,<\/h1>\n<ul>\n<li>In the &#8216;Dataflow&#8217; tab, click on &#8216;List jobs&#8217; tab.<\/li>\n<\/ul>\n<p><a href=\"https:\/\/m2msupport.net\/m2msupport\/wp-content\/uploads\/2021\/08\/list_dataflows.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-192957\" src=\"https:\/\/m2msupport.net\/m2msupport\/wp-content\/uploads\/2021\/08\/list_dataflows.png\" alt=\"\" width=\"851\" height=\"728\" srcset=\"https:\/\/m2msupport.net\/m2msupport\/wp-content\/uploads\/2021\/08\/list_dataflows.png 851w, https:\/\/m2msupport.net\/m2msupport\/wp-content\/uploads\/2021\/08\/list_dataflows-300x257.png 300w, https:\/\/m2msupport.net\/m2msupport\/wp-content\/uploads\/2021\/08\/list_dataflows-768x657.png 768w, https:\/\/m2msupport.net\/m2msupport\/wp-content\/uploads\/2021\/08\/list_dataflows-600x513.png 600w\" sizes=\"auto, (max-width: 851px) 100vw, 851px\" \/><\/a><\/p>\n<p>A GET request is made to GCP to get the list of dataflow jobs. I<\/p>\n<p>GET https:\/\/dataflow.googleapis.com\/v1b3\/projects\/second-inquiry-315605\/locations\/asia-east1\/jobs HTTP\/1.1<\/p>\n<p><strong>Server response to get the list of dataflow jobs.<\/strong><\/p>\n<p>{<\/p>\n<p>&#8220;jobs&#8221;: [<\/p>\n<p>{<\/p>\n<p>&#8220;id&#8221;: &#8220;2021-08-08_10_07_27-12462317361521435203&#8221;,<\/p>\n<p>&#8220;projectId&#8221;: &#8220;second-inquiry-315605&#8221;,<\/p>\n<p>&#8220;name&#8221;: &#8220;subscription_to_bq&#8221;,<\/p>\n<p>&#8220;type&#8221;: &#8220;JOB_TYPE_STREAMING&#8221;,<\/p>\n<p>&#8220;currentState&#8221;: &#8220;JOB_STATE_RUNNING&#8221;,<\/p>\n<p>&#8220;currentStateTime&#8221;: &#8220;2021-08-08T17:07:35.183260Z&#8221;,<\/p>\n<p>&#8220;createTime&#8221;: &#8220;2021-08-08T17:07:28.696060Z&#8221;,<\/p>\n<p>&#8220;location&#8221;: &#8220;asia-east1&#8221;,<\/p>\n<p>&#8220;jobMetadata&#8221;: {<\/p>\n<p>&#8220;sdkVersion&#8221;: {<\/p>\n<p>&#8220;version&#8221;: &#8220;2.29.0&#8221;,<\/p>\n<p>&#8220;versionDisplayName&#8221;: &#8220;Apache Beam SDK for Java&#8221;,<\/p>\n<p>&#8220;sdkSupportStatus&#8221;: &#8220;SUPPORTED&#8221;<\/p>\n<p>}<\/p>\n<p>},<\/p>\n<p>&#8220;startTime&#8221;: &#8220;2021-08-08T17:07:28.696060Z&#8221;<\/p>\n<p>},<\/p>\n<p>{<\/p>\n<p>&#8220;id&#8221;: &#8220;2021-08-08_08_10_53-4543477010421041882&#8221;,<\/p>\n<p>&#8220;projectId&#8221;: &#8220;second-inquiry-315605&#8221;,<\/p>\n<p>&#8220;name&#8221;: &#8220;topic_to_bq&#8221;,<\/p>\n<p>&#8220;type&#8221;: &#8220;JOB_TYPE_STREAMING&#8221;,<\/p>\n<p>&#8220;currentState&#8221;: &#8220;JOB_STATE_RUNNING&#8221;,<\/p>\n<p>&#8220;currentStateTime&#8221;: &#8220;2021-08-08T15:11:03.179279Z&#8221;,<\/p>\n<p>&#8220;createTime&#8221;: &#8220;2021-08-08T15:10:54.353075Z&#8221;,<\/p>\n<p>&#8220;location&#8221;: &#8220;asia-east1&#8221;,<\/p>\n<p>&#8220;jobMetadata&#8221;: {<\/p>\n<p>&#8220;sdkVersion&#8221;: {<\/p>\n<p>&#8220;version&#8221;: &#8220;2.29.0&#8221;,<\/p>\n<p>&#8220;versionDisplayName&#8221;: &#8220;Apache Beam SDK for Java&#8221;,<\/p>\n<p>&#8220;sdkSupportStatus&#8221;: &#8220;SUPPORTED&#8221;<\/p>\n<p>}<\/p>\n<p>},<\/p>\n<p>&#8220;startTime&#8221;: &#8220;2021-08-08T15:10:54.353075Z&#8221;<\/p>\n<p>},<\/p>\n<p>{<\/p>\n<p>&#8220;id&#8221;: &#8220;2021-08-08_07_59_39-14713810561228407080&#8221;,<\/p>\n<p>&#8220;projectId&#8221;: &#8220;second-inquiry-315605&#8221;,<\/p>\n<p>&#8220;name&#8221;: &#8220;topic_to_biquery&#8221;,<\/p>\n<p>&#8220;type&#8221;: &#8220;JOB_TYPE_STREAMING&#8221;,<\/p>\n<p>&#8220;currentState&#8221;: &#8220;JOB_STATE_CANCELLED&#8221;,<\/p>\n<p>&#8220;currentStateTime&#8221;: &#8220;2021-08-08T15:08:02.340401Z&#8221;,<\/p>\n<p>&#8220;createTime&#8221;: &#8220;2021-08-08T14:59:41.212809Z&#8221;,<\/p>\n<p>&#8220;location&#8221;: &#8220;asia-east1&#8221;,<\/p>\n<p>&#8220;jobMetadata&#8221;: {<\/p>\n<p>&#8220;sdkVersion&#8221;: {<\/p>\n<p>&#8220;version&#8221;: &#8220;2.29.0&#8221;,<\/p>\n<p>&#8220;versionDisplayName&#8221;: &#8220;Apache Beam SDK for Java&#8221;,<\/p>\n<p>&#8220;sdkSupportStatus&#8221;: &#8220;SUPPORTED&#8221;<\/p>\n<p>}<\/p>\n<p>},<\/p>\n<p>&#8220;startTime&#8221;: &#8220;2021-08-08T14:59:41.212809Z&#8221;<\/p>\n<p>},<\/p>\n<p>{<\/p>\n<p>&#8220;id&#8221;: &#8220;2021-07-27_05_02_22-11252401891348302536&#8221;,<\/p>\n<p>&#8220;projectId&#8221;: &#8220;second-inquiry-315605&#8221;,<\/p>\n<p>&#8220;name&#8221;: &#8220;job-1&#8221;,<\/p>\n<p>&#8220;type&#8221;: &#8220;JOB_TYPE_STREAMING&#8221;,<\/p>\n<p>&#8220;currentState&#8221;: &#8220;JOB_STATE_CANCELLED&#8221;,<\/p>\n<p>&#8220;currentStateTime&#8221;: &#8220;2021-08-03T13:27:38.522338Z&#8221;,<\/p>\n<p>&#8220;createTime&#8221;: &#8220;2021-07-27T12:02:25.532851Z&#8221;,<\/p>\n<p>&#8220;location&#8221;: &#8220;asia-east1&#8221;,<\/p>\n<p>&#8220;jobMetadata&#8221;: {<\/p>\n<p>&#8220;sdkVersion&#8221;: {<\/p>\n<p>&#8220;version&#8221;: &#8220;2.29.0&#8221;,<\/p>\n<p>&#8220;versionDisplayName&#8221;: &#8220;Apache Beam SDK for Java&#8221;,<\/p>\n<p>&#8220;sdkSupportStatus&#8221;: &#8220;SUPPORTED&#8221;<\/p>\n<p>}<\/p>\n<p>},<\/p>\n<p>&#8220;startTime&#8221;: &#8220;2021-07-27T12:02:25.532851Z&#8221;<\/p>\n<p>}<\/p>\n<p>]<\/p>\n<p>}<\/p>\n<p>You can also view the list of dataflow jobs in the Google console.<\/p>\n<p><a href=\"https:\/\/m2msupport.net\/m2msupport\/wp-content\/uploads\/2021\/08\/subscription4.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-192945\" src=\"https:\/\/m2msupport.net\/m2msupport\/wp-content\/uploads\/2021\/08\/subscription4.png\" alt=\"\" width=\"1636\" height=\"822\" srcset=\"https:\/\/m2msupport.net\/m2msupport\/wp-content\/uploads\/2021\/08\/subscription4.png 1636w, https:\/\/m2msupport.net\/m2msupport\/wp-content\/uploads\/2021\/08\/subscription4-300x151.png 300w, https:\/\/m2msupport.net\/m2msupport\/wp-content\/uploads\/2021\/08\/subscription4-768x386.png 768w, https:\/\/m2msupport.net\/m2msupport\/wp-content\/uploads\/2021\/08\/subscription4-1024x515.png 1024w, https:\/\/m2msupport.net\/m2msupport\/wp-content\/uploads\/2021\/08\/subscription4-600x301.png 600w\" sizes=\"auto, (max-width: 1636px) 100vw, 1636px\" \/><\/a><\/p>\n<div class=\"video-responsive\"><iframe loading=\"lazy\" id=\"youTubePlayer\" src=\"https:\/\/www.youtube.com\/embed\/WdQ_jzrhPgU?hd=1\" width=\"750\" height=\"421\" frameborder=\"0\" allowfullscreen=\"allowfullscreen\"><\/iframe><\/div>\n","protected":false},"excerpt":{"rendered":"<p>Google Cloud Dataflow is data processing service that can be used for streaming and batch applications. Users can setup pipelines in Dataflow to integrate and process large datasets. With pub\/sub, users can setup dataflow pipelines to write messages from a &hellip; <a href=\"https:\/\/m2msupport.net\/m2msupport\/get-a-list-of-dataflow-jobs-in-google-cloud-platform-gcp\/\">Continue reading <span class=\"meta-nav\">&rarr;<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"iot_tutorial_template.php","meta":{"footnotes":""},"class_list":["post-192956","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"https:\/\/m2msupport.net\/m2msupport\/wp-json\/wp\/v2\/pages\/192956","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/m2msupport.net\/m2msupport\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/m2msupport.net\/m2msupport\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/m2msupport.net\/m2msupport\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/m2msupport.net\/m2msupport\/wp-json\/wp\/v2\/comments?post=192956"}],"version-history":[{"count":1,"href":"https:\/\/m2msupport.net\/m2msupport\/wp-json\/wp\/v2\/pages\/192956\/revisions"}],"predecessor-version":[{"id":192958,"href":"https:\/\/m2msupport.net\/m2msupport\/wp-json\/wp\/v2\/pages\/192956\/revisions\/192958"}],"wp:attachment":[{"href":"https:\/\/m2msupport.net\/m2msupport\/wp-json\/wp\/v2\/media?parent=192956"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}