Get a list of dataflow jobs in Google Cloud Platform (GCP)
Google Cloud Dataflow is data processing service that can be used for streaming and batch applications. Users can setup pipelines in Dataflow to integrate and process large datasets.
With pub/sub, users can setup dataflow pipelines to write messages from a pub/sub topic or subscription to a BigQuery table.
IoT Cloud Tester application provides an easy interface to get a list of dataflow jobs in Google Cloud Platform.
To get a list of dataflow jobs in Google Cloud,
- In the 'Dataflow' tab, click on 'List jobs' tab.
A GET request is made to GCP to get the list of dataflow jobs. I
GET https://dataflow.googleapis.com/v1b3/projects/second-inquiry-315605/locations/asia-east1/jobs HTTP/1.1
Server response to get the list of dataflow jobs.
{
"jobs": [
{
"id": "2021-08-08_10_07_27-12462317361521435203",
"projectId": "second-inquiry-315605",
"name": "subscription_to_bq",
"type": "JOB_TYPE_STREAMING",
"currentState": "JOB_STATE_RUNNING",
"currentStateTime": "2021-08-08T17:07:35.183260Z",
"createTime": "2021-08-08T17:07:28.696060Z",
"location": "asia-east1",
"jobMetadata": {
"sdkVersion": {
"version": "2.29.0",
"versionDisplayName": "Apache Beam SDK for Java",
"sdkSupportStatus": "SUPPORTED"
}
},
"startTime": "2021-08-08T17:07:28.696060Z"
},
{
"id": "2021-08-08_08_10_53-4543477010421041882",
"projectId": "second-inquiry-315605",
"name": "topic_to_bq",
"type": "JOB_TYPE_STREAMING",
"currentState": "JOB_STATE_RUNNING",
"currentStateTime": "2021-08-08T15:11:03.179279Z",
"createTime": "2021-08-08T15:10:54.353075Z",
"location": "asia-east1",
"jobMetadata": {
"sdkVersion": {
"version": "2.29.0",
"versionDisplayName": "Apache Beam SDK for Java",
"sdkSupportStatus": "SUPPORTED"
}
},
"startTime": "2021-08-08T15:10:54.353075Z"
},
{
"id": "2021-08-08_07_59_39-14713810561228407080",
"projectId": "second-inquiry-315605",
"name": "topic_to_biquery",
"type": "JOB_TYPE_STREAMING",
"currentState": "JOB_STATE_CANCELLED",
"currentStateTime": "2021-08-08T15:08:02.340401Z",
"createTime": "2021-08-08T14:59:41.212809Z",
"location": "asia-east1",
"jobMetadata": {
"sdkVersion": {
"version": "2.29.0",
"versionDisplayName": "Apache Beam SDK for Java",
"sdkSupportStatus": "SUPPORTED"
}
},
"startTime": "2021-08-08T14:59:41.212809Z"
},
{
"id": "2021-07-27_05_02_22-11252401891348302536",
"projectId": "second-inquiry-315605",
"name": "job-1",
"type": "JOB_TYPE_STREAMING",
"currentState": "JOB_STATE_CANCELLED",
"currentStateTime": "2021-08-03T13:27:38.522338Z",
"createTime": "2021-07-27T12:02:25.532851Z",
"location": "asia-east1",
"jobMetadata": {
"sdkVersion": {
"version": "2.29.0",
"versionDisplayName": "Apache Beam SDK for Java",
"sdkSupportStatus": "SUPPORTED"
}
},
"startTime": "2021-07-27T12:02:25.532851Z"
}
]
}
You can also view the list of dataflow jobs in the Google console.
