Custom search attribute doesn't work with temporal server start-dev

[karney]

Started using temporal server start-dev for local development and was able to successfully add search attributes. The search attributes show up when I run temporal operator search-attribute list but when the workflow attempts to call Workflow.upsertSearchAttributes it gets an error:

io.grpc.StatusRuntimeException: INVALID_ARGUMENT: BadSearchAttributes: Namespace my-namespace has no mapping defined for search attribute Type

(my-namespace is the name of the namespace and the name of the attribute is called “Type”).

I also noticed that in the Temporal UI when I look at the Search Attributes list in the namespace, instead of the names I gave them, they just show up as “Keyword01”, “Keyword02”, etc.

I’ve tried stopping and restarting the server to no-avail.

[karney]

Seems like using tctl admin cluster asa to register the search attributes works, but using the temporal operation command doesn’t. The incorrect search attribute names in the UI persist regardless.

[taonic]

Hi Karney,

I can’t seem to reproduce this issue with Temporal CLI 0.9.0.

To eliminate a possibility, can you confirm you are adding the search attribute to the correct namespace my-namespace and you can see it from temporal operator search-attribute list --namespace my-namespace?

[karney]

Earlier, I relied on exporting the environment variable TEMPORAL_NAMESPACE to set the namespace.

As a test, I removed the attribute using tctl, then re-added it using temporal specifying the namespace option explicitly and it now works. Perhaps there’s an issue with the env variable not getting picked up, I haven’t tested that.

The output from search-attribute list looks the same as previously, so it’s not possible to see any difference there.

The Temporal UI never the less doesn’t seem to display the search attributes for the namespace correctly like the CLI does.

I am on the same version of temporal as you

temporal version 0.9.0 (server 1.20.2) (ui 2.15.0)

[taonic]

The namespace env var has changed to TEMPORAL_CLI_NAMESPACE in Temporal CLI Temporal CLI command reference | Temporal Documentation

I can see the same UI issue as you reported. If no one else follow up on this thread I can take a look at the code tomorrow morning. It looks like a bug at this point.

[karney]

Might want to also update the output for temporal env list --help which currently mentions $TEMPORAL_NAMESPACE

[avarabyeu]

Can confirm - UI and temporal CLI (as well as API using golang SDK) show different list of attributes. Custom attributes seem to be created successfully over CLI or API, but adding them to workflow results in Namespace default has no mapping defined for search attribute <attribute name>
Temporal CLI 0.9.0, SQLLite persistance.
Attempting to set/unset system.forceSearchAttributesCacheRefreshOnRead has no affect

[petersamokhin]

Hello there! I face the same issue with typescript SDK (upsertSearchAttributes) & typescript worker, while creating the attributes correctly and seeing them in the temporal web UI and using cli (temporal operator search-attribute list).

{
  "message": "BadSearchAttributes: Namespace MY_NAMESPACE has no mapping defined for search attribute MY_SEARCH_ATTRIBUTE",
  "source": "",
  "stackTrace": "",
  "encodedAttributes": null,
  "cause": null,
  "serverFailureInfo": {
    "nonRetryable": true
  }
}

Binary Checksum: @temporalio/worker@1.7.4+e7abfc11f1d9da4c176cc2b7ca184b1e81106f264157a84eaa4801078763688a

[taonic]

@karney Thanks for pointing out the misleading env var names in CLI’s prompt. I’ll follow up it from my side. This has been fixed here and should be included in the next release with updated document too.

@avarabyeu and @petersamokhin

Firstly, I’d like to acknowledge the defect in the UI where the custom search attributes are not showing up accordingly:

[Bug] Custom search attributes not showing up in UI

Now, based on what Karney described above, as well as my earlier test, the error message such as Namespace default has no mapping defined for search attribute <attribute name> may have something to do with different namespaces being used while creating the custom attribute and consuming/updating it. Note, custom attributes are scoped to namespaces. So it’d be great if you could help me to double check this scenario first. Thanks!

[avarabyeu]

I tend to think that issue is related to back-end, not to UI. Mostly, because we cant create workflow with just created search attributes (as well as upsert them to the workflow).
As far namespace part - the UI says i’ve got only “default”. I see “default” and “temporal-system” in DB and all custom attributes are created with passing “default” explicitly.
Is there anything else i can check in SQLLite to help you with figuring out?

[taonic]

Let’s eliminate the SDK for now. Can you share the result of these cli commands? e.g.

$ temporal operator search-attribute create --name MyKeyword1 --type Keyword
Search attributes have been added

$ temporal workflow start --type test --task-queue test --search-attribute MyKeyword1=\"abc\
Running execution:
  WorkflowId  95cd7128-1b3a-49b3-af65-edf2532a83c5
  RunId       f6660b6a-8dd3-4a72-8440-7486cc54f3e0
  Type        test
  Namespace   default
  TaskQueue   test
  Args        []

$ temporal -v
temporal version 0.9.0 (server 1.20.2) (ui 2.15.0)

FWIW, this is my server command:

temporal server start-dev -db-filename db

[karney]

I was able to register the values in the non-default namespace after specifying it explicitly using the namespace option in the command. The initial problem is likely because of the env var not having any effect.

[avarabyeu]

Search attributes have been added

Running execution:
  WorkflowId  2d471a2b-8b6a-4e65-af58-0f2e9c757fcc
  RunId       270c2058-2fb3-4291-af69-883218aba5e1
  Type        test
  Namespace   default
  TaskQueue   test
  Args        []

so it works perfectly well. now it gets a bit interesting. I tried to do the same with my code (adding an attribute via temporal client and then starting the workflow) and it doesn’t work. Probably, something is related to caching or asynchronous nature of adding new search attribute. The temporal is run as:
temporal server start-dev -n default -db-filename db --dynamic-config-value system.forceSearchAttributesCacheRefreshOnRead=true

[avarabyeu]

Well, it looks like there are two separate issues - with UI showing incorrect list of attributes and on my end with caches.
So, my sequence of actions:

1. Start the dev server, wait for cluster to become healthy
2. Add new search attributes over client SDK
3. Execute list attributes, make sure new attributes are returned
4. Start new workflow that uses just created attributes (also over client SDK)
   As result, search attributes aren’t available for the workflow and the result is Namespace default has no mapping defined for search attribute)
   Still, if i add some 20 secs wait in between of (3) and (4) - it works perfectly fine. The same reproduces regardless the value system.forceSearchAttributesCacheRefreshOnRead set to

[avarabyeu]

@taonic Is there any way to make sure that custom search attributes are available for use by ExecuteWorkflow? There is still some time in between of ListSearchAttributes returns new attributes and ExecuteWorkflow stops responding with the error that attribute isn’t defined

[taonic]

Turns out it’s a regression: Regression: namespace specific search attributes have registration delay with `system.forceSearchAttributesCacheRefreshOnRead=true` · Issue #4017 · temporalio/temporal · GitHub

[avarabyeu]

@taonic thank you for sharing the update on this!

[CliffSun]

Am I right in saying that there’s still an issue with displaying the custom search attributes in the UI? This is what I’ve still noticed when starting a temporal dev server using the latest Temporal CLI. The actual search attribute is registered and creating workflows that set that search attribute succeeds, so it sounds like a bug on the UI side.