To setup a dataflow to write messages from pub\/sub topic to BigQuery table,<\/h1>\n\n- In the ‘Dataflow’ tab, click on ‘Create Job’ tab.<\/li>\n
- Enter the job name<\/li>\n
- Select ‘Pub\/Sub Topic to BigQuery’ option<\/li>\n
- Get the list of topics and select one<\/li>\n
- Get the available cloud storage buckets for the project and select one.<\/li>\n
- Enter the file name. This file is used by the dataflow.<\/li>\n
- Setup the Dataset and Table to be use to write the messages from the pub\/sub topic. Note that the table schema should match the pub\/sub topic message structure.<\/li>\n<\/ul>\n
![\"\"](\"https:\/\/m2msupport.net\/m2msupport\/wp-content\/uploads\/2021\/08\/dataflow1.png\")
<\/a><\/p>\nDataflow Job ‘topic_to_bq’ is created immediately with pending status.<\/p>\n
![\"\"](\"https:\/\/m2msupport.net\/m2msupport\/wp-content\/uploads\/2021\/08\/dataflow2.png\")
<\/a><\/p>\nA post request is made to GCP to create the dataflow job. In this case, we’re using the pre-build template PubSub_to_BigQuery to write pub\/sub topic messages to BigQuery.<\/p>\n
POST https:\/\/dataflow.googleapis.com\/v1b3\/projects\/second-inquiry-315605\/locations\/asia-east1\/templates:launch?gcsPath=gs:\/\/dataflow-templates\/latest\/PubSub_to_BigQuery HTTP\/1.1<\/p>\n
Server response for job creation.<\/strong><\/p>\n{“jobName”:”topic_to_bq”,”environment”:{“tempLocation”:”gs:\/\/my-iot-bucket-5\/temp_file_for_dataflow”,”additionalExperiments”:[],”bypassTempDirValidation”:false,”ipConfiguration”:”WORKER_IP_UNSPECIFIED”},”parameters”:{“outputTableSpec”:”second-inquiry-315605:device_data.environment”,”inputTopic”:”projects\/second-inquiry-315605\/topics\/environment”}}<\/p>\n
{<\/p>\n
“job”: {<\/p>\n
“id”: “2021-08-08_08_10_53-4543477010421041882”,<\/p>\n
“projectId”: “second-inquiry-315605”,<\/p>\n
“name”: “topic_to_bq”,<\/p>\n
“type”: “JOB_TYPE_STREAMING”,<\/p>\n
“currentStateTime”: “1970-01-01T00:00:00Z”,<\/p>\n
“createTime”: “2021-08-08T15:10:54.353075Z”,<\/p>\n
“location”: “asia-east1”,<\/p>\n
“startTime”: “2021-08-08T15:10:54.353075Z”<\/p>\n
}<\/p>\n
}<\/p>\n
The newly created dataflow job can be viewed in the Google console.<\/p>\n
![\"\"](\"https:\/\/m2msupport.net\/m2msupport\/wp-content\/uploads\/2021\/08\/dataflow3.png\")
<\/a><\/p>\nNow let us see the dataflow in action. We’ll have a device publish message to that topic and verify that the data is written to the BigQuery table.<\/p>\n
Below device ‘dev_23992’ is publishing to topic ‘environment’. Above we have setup a dataflow job to write that topic messages to the ‘environment’ table in the device_data dataset in BigQuery.<\/p>\n
Dataflow Job ‘topic_to_bq’ is created immediately with pending status.<\/p>\n A post request is made to GCP to create the dataflow job. In this case, we’re using the pre-build template PubSub_to_BigQuery to write pub\/sub topic messages to BigQuery.<\/p>\n POST https:\/\/dataflow.googleapis.com\/v1b3\/projects\/second-inquiry-315605\/locations\/asia-east1\/templates:launch?gcsPath=gs:\/\/dataflow-templates\/latest\/PubSub_to_BigQuery HTTP\/1.1<\/p>\n Server response for job creation.<\/strong><\/p>\n {“jobName”:”topic_to_bq”,”environment”:{“tempLocation”:”gs:\/\/my-iot-bucket-5\/temp_file_for_dataflow”,”additionalExperiments”:[],”bypassTempDirValidation”:false,”ipConfiguration”:”WORKER_IP_UNSPECIFIED”},”parameters”:{“outputTableSpec”:”second-inquiry-315605:device_data.environment”,”inputTopic”:”projects\/second-inquiry-315605\/topics\/environment”}}<\/p>\n {<\/p>\n “job”: {<\/p>\n “id”: “2021-08-08_08_10_53-4543477010421041882”,<\/p>\n “projectId”: “second-inquiry-315605”,<\/p>\n “name”: “topic_to_bq”,<\/p>\n “type”: “JOB_TYPE_STREAMING”,<\/p>\n “currentStateTime”: “1970-01-01T00:00:00Z”,<\/p>\n “createTime”: “2021-08-08T15:10:54.353075Z”,<\/p>\n “location”: “asia-east1”,<\/p>\n “startTime”: “2021-08-08T15:10:54.353075Z”<\/p>\n }<\/p>\n }<\/p>\n The newly created dataflow job can be viewed in the Google console.<\/p>\n Now let us see the dataflow in action. We’ll have a device publish message to that topic and verify that the data is written to the BigQuery table.<\/p>\n Below device ‘dev_23992’ is publishing to topic ‘environment’. Above we have setup a dataflow job to write that topic messages to the ‘environment’ table in the device_data dataset in BigQuery.<\/p>\n<\/a><\/p>\n<\/a><\/p>\n<\/a><\/p>\n
![\"\"](\"https:\/\/m2msupport.net\/m2msupport\/wp-content\/uploads\/2021\/08\/dataflow4.png\")
<\/a><\/p>\n![\"\"](\"https:\/\/m2msupport.net\/m2msupport\/wp-content\/uploads\/2021\/08\/dataflow5.png\")
<\/a><\/p>\n