To setup a dataflow to write messages from pub\/sub topic to\u00a0 BigQuery table,<\/h1>\n\n- In the ‘Dataflow’ tab, click on ‘Create Job’ tab.<\/li>\n
- Enter the job name<\/li>\n
- Select ‘Pub\/Sub Topic to BigQuery’ option<\/li>\n
- Get the list of topics and select one<\/li>\n
- Get the available cloud storage buckets for the project and select one.<\/li>\n
- Enter the file name. This file is used by the dataflow.<\/li>\n
- Setup the Dataset and Table to be use to write the messages from the pub\/sub topic. Note that the table schema should match the pub\/sub topic message structure.<\/li>\n<\/ul>\n
![\"\"](\"https:\/\/m2msupport.net\/m2msupport\/wp-content\/uploads\/2021\/08\/subscription1.png\")
<\/a><\/p>\nDataflow Job ‘subscription_to_bq’ is created immediately with pending status.<\/p>\n
![\"\"](\"https:\/\/m2msupport.net\/m2msupport\/wp-content\/uploads\/2021\/08\/subscription3.png\")
<\/a><\/p>\nA post request is made to GCP to create the dataflow job. In this case, we’re using the pre-build template\u00a0PubSub_Subscription_to_BigQuery to write pub\/sub subscription messages to BigQuery.<\/p>\n
POST https:\/\/dataflow.googleapis.com\/v1b3\/projects\/second-inquiry-315605\/locations\/asia-east1\/templates:launch?gcsPath=gs:\/\/dataflow-templates\/latest\/PubSub_Subscription_to_BigQuery HTTP\/1.1<\/p>\n
Server response for job creation.<\/strong><\/p>\n{<\/p>\n
“job”: {<\/p>\n
“id”: “2021-08-08_10_07_27-12462317361521435203”,<\/p>\n
“projectId”: “second-inquiry-315605”,<\/p>\n
“name”: “subscription_to_bq”,<\/p>\n
“type”: “JOB_TYPE_STREAMING”,<\/p>\n
“currentStateTime”: “1970-01-01T00:00:00Z”,<\/p>\n
“createTime”: “2021-08-08T17:07:28.696060Z”,<\/p>\n
“location”: “asia-east1”,<\/p>\n
“startTime”: “2021-08-08T17:07:28.696060Z”<\/p>\n
}<\/p>\n
}<\/p>\n
The newly created dataflow job can be viewed in the Google console.<\/p>\n
![\"\"](\"https:\/\/m2msupport.net\/m2msupport\/wp-content\/uploads\/2021\/08\/subscription4.png\")
<\/a><\/p>\nNow let us see the dataflow in action. We’ll have a device publish message to that subscription and verify\u00a0 that the data is written to the BigQuery table.<\/p>\n
Below device ‘dev_23992’ is publishing to topic ‘environment’. We have a subscription ‘subscription_628331’ to topic ‘environment’. Above we also have setup a dataflow job to write that subscription messages to the ‘environment_subscription’ table in the device_data dataset in BigQuery.<\/p>\n
Dataflow Job ‘subscription_to_bq’ is created immediately with pending status.<\/p>\n A post request is made to GCP to create the dataflow job. In this case, we’re using the pre-build template\u00a0PubSub_Subscription_to_BigQuery to write pub\/sub subscription messages to BigQuery.<\/p>\n POST https:\/\/dataflow.googleapis.com\/v1b3\/projects\/second-inquiry-315605\/locations\/asia-east1\/templates:launch?gcsPath=gs:\/\/dataflow-templates\/latest\/PubSub_Subscription_to_BigQuery HTTP\/1.1<\/p>\n Server response for job creation.<\/strong><\/p>\n {<\/p>\n “job”: {<\/p>\n “id”: “2021-08-08_10_07_27-12462317361521435203”,<\/p>\n “projectId”: “second-inquiry-315605”,<\/p>\n “name”: “subscription_to_bq”,<\/p>\n “type”: “JOB_TYPE_STREAMING”,<\/p>\n “currentStateTime”: “1970-01-01T00:00:00Z”,<\/p>\n “createTime”: “2021-08-08T17:07:28.696060Z”,<\/p>\n “location”: “asia-east1”,<\/p>\n “startTime”: “2021-08-08T17:07:28.696060Z”<\/p>\n }<\/p>\n }<\/p>\n The newly created dataflow job can be viewed in the Google console.<\/p>\n Now let us see the dataflow in action. We’ll have a device publish message to that subscription and verify\u00a0 that the data is written to the BigQuery table.<\/p>\n Below device ‘dev_23992’ is publishing to topic ‘environment’. We have a subscription ‘subscription_628331’ to topic ‘environment’. Above we also have setup a dataflow job to write that subscription messages to the ‘environment_subscription’ table in the device_data dataset in BigQuery.<\/p>\n<\/a><\/p>\n<\/a><\/p>\n<\/a><\/p>\n
![\"\"](\"https:\/\/m2msupport.net\/m2msupport\/wp-content\/uploads\/2021\/08\/subscription5.png\")
<\/a><\/p>\n![\"\"](\"https:\/\/m2msupport.net\/m2msupport\/wp-content\/uploads\/2021\/08\/subscription2.png\")
<\/a><\/p>\n